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Results  in  Brief 

Advanced  Combat  Helmet  Technical  Assessment 


May  29,  2013 

Objective 

Our  objective  was  to  assess  the  methods  and  technical 
rationale  in  developing  the  Advanced  Combat  Helmet 
(ACH)  testing  protocols  issued  by  the  Office  of  the 
Director,  Operational  Test  and  Evaluation  (DOT&E)  to 
determine  whether  the  test  protocols  are  appropriate 
for  the  ACH.  Specifically,  our  assessment  focused  on 
the  First  Article  Testing  (FAT)  Resistance  to  Penetration 
(RTP)  requirement  of  at  least  90  percent  Probability 
of  no  Penetration  (P(nP))  with  90  percent  Confidence 
Level,  commonly  termed  the  “90/90  standard,"  and 
the  Lot  Acceptance  Testing  (LAT)  RTP  requirement  of 
4  percent  Acceptable  Quality  Level  (AQL).  Lastly,  we 
assessed  the  participation  of  various  stakeholders  and 
industry  experts  such  as  active  ACH  manufacturers  and 
test  facilities. 

Findings 

After  reviewing  the  methods  and  technical  rationale  in 
developing  the  helmet  RTP  requirements,  we  found  the 
following. 

A.  The  DOT&E  test  protocol  for  the  ACH  adopts  a 
statistically  principled  approach  and  represents 
an  improvement  from  the  legacy  test  protocol  with 
regard  to  increased  sample  size.  However,  future 
protocol  revisions  necessitate  further  refinement  by 
anchoring  the  RTP  requirements  to  helmet  specific 
empirical  data  such  as  manufacturing  capabilities 
and  test  performance. 

B.  The  DOT&E  LAT  protocol  is  an  improvement  from 
the  legacy  LAT  and  adopts  a  widely  established 
and  industrially  accepted  American  National 
Standards  Institute  (ANSI  Zl.4-2008,  Sampling 


Findings  Continued 

Procedures  and  Tables  for  Inspection  by  Attributes).  In 
selecting  the  LAT  RTP  requirement  of  4  percent  AQL, 
DOT&E  considered  the  government  risk  of  accepting 
underperforming  helmets,  manufacturer  risk  of  failing 
LAT  with  acceptable  helmets,  and  historical  LAT  data. 
However,  DOT&E  did  not  consider  selecting  an  AQL  that 
was  based  on  the  safety  criticality  of  the  helmet. 

C.  In  accordance  with  authorizing  statutes,  DOT&E  has 
the  authority  to  establish  test  standards  for  personnel 
protective  equipment  such  as  the  ACH.  However,  despite 
the  significance  and  broad  impact  of  these  protocols, 
DOT&E  did  not  explicitly  consult  with  heads  of  the 
Military  Departments  to  provide  them  an  opportunity 
to  comment  on  new  or  changed  test  protocols  and  did 
not  adequately  document  the  adjudication  of  inputs 
provided  by  program  offices  and  subject  matter  experts 
in  the  staffing  process.  The  program  office  also  did  not 
solicit  comments  on  the  helmet  test  protocols  with  the 
helmet  vendors  and  Defense  Contract  Management 
Agency  (DCMA). 

D.  The  FAT  RTP  acceptance  was  based  on  an  aggregate  of 
all  test  outcomes  under  varying  conditions  to  achieve 
90/90,  or  17  penetrations  out  of  240  shots.  This  could 
result  in  passing  FAT,  despite  test  results  showing 
clusters  of  failures  for  a  unique  helmet  size  or  in  a 
particular  test  environment. 

Recommendations 

During  the  course  of  this  assessment,  DOT&E  and  the 
Army's  Program  Executive  Office  (PEO)  Soldier  were  very 
responsive  and  have  already  committed  to  address  most  of 
the  findings  identified.  Our  recommendations  pertaining  to 
the  findings  are  detailed  on  the  next  page. 


Visit  us  on  the  web  at  www.dodig.mil 
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Recommendations  Continued 

Recommendation  for  Finding  A.  Origin  of 
Resistance  to  Penetration  Requirements 

We  recommend  that  DOT&E  and  PEO  Soldier  fully  characterize 
the  performance  of  all  helmet  designs  included  in  the  combat 
helmet  test  protocols.  Performance  characterization  should 
consider  threat,  historical  test  data,  prototype  test  data,  and 
manufacturing  capabilities.  Based  on  helmet  performance 
characterizations,  DOT&E  and  PEO  Soldier  should  determine 
if  modification  to  the  FAT  and  LAT  protocols  are  appropriate. 

Recommendation  for  Finding  B.  Acceptable 
Quality  Level  Based  on  Safety  Criticality 

We  recommend  that  in  addition  to  considerations  made 
to  Government  risk,  manufacturer  risk,  and  historical  LAT 
data,  DOT&E  should  also  consider  the  safety  criticality  of  the 
helmet  in  establishing  the  AQL  requirement  for  LAT  RTP  in 
future  protocol  revisions. 

Recommendation  for  Finding  C.  Coordination  of 
Helmet  Test  Protocols 

Due  to  the  significance  and  broad  impact  of  these  protocols, 
we  recommend  that  DOT &E  ensure  the  affected  organizations, 
including  heads  of  the  Military  Departments,  are  consulted  in 
developing  the  protocols  and  have  an  opportunity  to  provide 
input  on  new  or  changed  test  protocols.  DOT&E  should  also 
ensure  that  inputs  received  are  documented  and  adjudicated. 

We  recommend  that  PEO  Soldier  solicit  input  from  helmet 
vendors  and  DCMA  on  new  or  changed  test  protocols. 
PEO  Soldier  should  also  ensure  that  inputs  received  are 
documented  and  adjudicated. 


We  recommend  that  DOT&E  include  an  explicit  statement  in 
future  protocol  revisions  that  allows  program  managers  to 
modify  the  test  protocols,  provided  that  program  managers 
submit  a  well-justified  request  for  approval. 

Recommendation  for  Finding  D.  Evaluation  of 
Aggregate  Test  Results 

We  recommend  that  DOT&E  and  PEO  Soldier  describe  the 
method  of  identifying  and  addressing  statistically  significant 
differences  in  performance  due  to  environmental  conditions, 
helmet  sizes,  shot  locations,  and  different  vendor  designs  for 
all  FAT  results  under  the  DOT&E  helmet  test  protocol. 

We  recommend  that  PEO  Soldier  and  Defense  Logistics 
Agency  (DLA)  consider  the  contractual  implications  of  not 
fielding  an  ACH  design  that  passes  FAT  yet  shows  significant 
clusters  of  inadequate  performance. 

Overall  DOT&E  and  PEO  Soldier 
Comment 

DOT&E  and  Program  Executive  Office  (PEO)  Soldier  have 
reviewed  the  referenced  report  and  agree  with  the  findings 
and  recommendations. 

Overall  DoD  OIG  Response 

DOT&E  and  PEO  Soldier's  comments  were  fully  responsive. 
We  reviewed  their  response  and  have  additional  points 
for  consideration.  However,  no  additional  comments  are 
required. 
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4800  MARK  CENTER  DRIVE 
ALEXANDRIA,  VIRGINIA  22350-1500 


INSPECTOR  GENERAL 
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MEMORANDUM  FOR  DIRECTOR,  OPERATIONAL  TEST  AND  EVALUATION 

U.S.  ARMY  PROGRAM  EXECUTIVE  OFFICE,  SOLDIER 

SUBJECT:  The  Advanced  Combat  Helmet  (ACH)  Technical  Assessment  (Project 
No.D2012-DT0TAD-0007.000) 

This  final  report  is  provided  for  information  and  use.  On  July  13,  2012,  the  DoD  Office 
of  Inspector  General  (OIG)  announced  the  ACH  Technical  Assessment  in  response  to  a 
request  from  Representative  Louise  M.  Slaughter  to  review  the  current  testing  protocols 
for  ACHs.  During  the  assessment,  we  found  that  the  helmet  test  protocols  are  an 
improvement  from  legacy  protocols.  The  first  article  test  protocol  adopts  a  statistically 
principled  approach  and  the  lot  acceptance  test  protocol  adopts  a  widely  established  and 
industrially  accepted  sampling  procedure.  We  provided  our  findings  and 
recommendations  for  review  and  determined  that  your  comments  were  fully  responsive. 

We  appreciate  the  courtesies  extended  to  the  staff.  For  additional  information  on  this 
report,  please  contact  A1  Dopita  at  (703)  699-0220  (DSN  664-0220),  or 
alois.dopita@dodig.mil. 


Randolph  Stone 
Deputy  Inspector  General 
Policy  and  Oversight 
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Introduction 


Objective 

This  report  is  in  response  to  a  request  from  Representative  Louise  M.  Slaughter,  which  can 
be  found  in  Appendix  B.  Representative  Slaughter  requested  on  June  26,  2012,  that  the 
DoD  Office  of  Inspector  General  (OIG)  review  the  current  testing  protocols  for  Advanced 
Combat  Helmets  (ACHs),  expressing  concerns  that: 

...  recent  modification  to  the  standard  for  ballistic  testing  for  ACHs 
would  allow  up  to  17  penetrations  out  of  240  test  shots.  The  re¬ 
sult  of  that  standard  is  a  90%  probability  of  no  perforation  [pen¬ 
etration]  with  90%  confidence  during  first  article  testing.  I  am 
concerned  that  a  standard  that  allows  17  penetrations  puts  con¬ 
tractors  above  the  need  to  produce  effective  and  consistent  equip¬ 
ment  for  the  men  and  women  of  the  military. 

While  procurement  standards  have  improved  in  recent  years,  the 
current  standard  poses  an  unacceptably  high  risk  for  equipment 
intended  to  protect  American  lives. 

Our  objective  was  to  assess  methods  and  technical  rationale  in  developing  ACH  testing 
protocols  issued  by  the  Office  of  the  Director,  Operational  Test  and  Evaluation  (DOT&E)1 
to  determine  whether  the  test  requirements  are  appropriate  for  the  ACH.  Specifically, 
our  assessment  focused  on  First  Article  Testing  (FAT)  Resistance  to  Penetration  (RTP) 
requirement  of  least  90  percent  Probability  of  no  Penetration  (P(nP))  with  90  percent 
Confidence  Level,  commonly  termed  the  "90/90  standard,"  and  Lot  Acceptance  Testing 
(LAT)  RTP  requirement  of  4  percent  Acceptable  Quality  Level  (AQL).  We  assessed  the 
participation  of  various  stakeholders  and  industry  experts,  including  the  DOT&E-man- 
dated  test  facility  and  ACH  manufacturers.  We  also  evaluated  the  RTP  requirements  of 
the  legacy  ACH  FAT  and  LAT  protocols  for  comparison.  (See  Appendix  A  for  discussion  of 
our  scope  and  methodology) 


1  Protocols  reviewed  were: 

"Military  Combat  Helmet  Standard  for  Ballistic  Testing"  issued  December  7 ,  2010 
"Military  Combat  Helmet  Standard  for  Ballistic  Testing"  issued  September  20,  2011 
"Military  Combat  Helmet  Standard  for  Ballistic  Lot  Acceptance  Testing"  issued  January  19,  2012 
"Military  Combat  Helmet  Standard  for  Ballistic  Lot  Acceptance  Testing"  issued  May  4,  2012 
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Background 

Advanced  Combat  Helmet 

The  ACH  replaces  the  Personnel  Armor  System,  Ground  Troops  (PASGT}  Helmet  for 
general  use  by  the  U.S.  Army.  The  ACH  consists  of  a  finished  ballistic  protective  shell, 
pad  suspension  system,  a  retention  system  (chin  strap/nape  strap},  cover,  and  eyewear 
retention  strap.  The  Acquisition  Decision  Memorandum  (ADM]  for  the  ACH  was  approved 
on  January  8,  2003,  with  the  goal  of  modifying  the  PASGT  Helmet  outer  shell  geometry, 
while  at  the  same  time  applying  material  technology  advancements  to  reduce  the  weight 
and  maintain  protection  against  fragmentation  of  9mm  small  arms  munitions. 

Figure  1.  Combat  Helmet  Design  Evolution 


Light  Weight  ACH  Solicitation 
Released 

8%  Reduction  in  Weight 


UCP  caprint  with  IFF  introduced 


ECH  w/  increased 
protection  against 
"select  small  arms 
threats" 


(initial) 

Helmet  weight 
Decrease  from 
3.5  lb  to  3.0  lb 
(approx.) 

— v' 


Pad  improvement 
impact  level  increase 
to  150g  max  at  10  fps 


PASGT  Helmet 


Figure  1  provides  the  evolution  of  combat  helmet  design  and  depicts  that  initial  fielding 
of  the  ACH  began  in  2002.  Pad  improvements  occurred  in  2006  and  2007.  For  example, 
helmet  sensors  were  developed  to  detect  when  a  helmet  has  been  subjected  to  a  force 
strong  enough  to  cause  brain  injury.  More  recent  efforts  are  underway  to  reduce  the 
weight  of  the  ACH  and  develop  a  helmet  with  increased  protection  against  select  small 
arms  threats. 

Program  Executive  Office  (PEO]  Soldier,  Product  Manager/Soldier  Protective  Equipment 
(PM/SPE],  is  the  program  office  responsible  for  developing  and  managing  the  ACH 
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product  specification.  PEO  Soldier  was  responsible  for  the  initial  fielding  of  the  ACH  using 
equipping  funds  beginning  in  August  2005.  Sustainment  activities  began  in  November 
2008  when  Defense  Logistics  Agency  Troop  Support  (DLA  TS)  became  responsible  for 
ACH  procurements.  However,  PEO  Soldier  retains  responsibility  for  life  cycle  management 
of  the  system,  including  the  product  description  and  technical  specifications.  PEO  Solider 
also  assists  DLA  TS  in  evaluating  vendors  for  sustainment  procurements. 

There  are  currently  two  active  ACH  sustainment  bridge  contracts  to  procure  ACHs  with 
design  specifications  detailed  in  Contract  Purchase  Description  "CO/PD-05-04,"  October 
30,  2007  (most  recently  updated  on  March  24,  2009).  The  first  is  a  bridge  contract  to 
Gentex  Corporation  (contract  number  SPM1C1-12-C0016)  and  the  second  to  Mine  Safety 
Appliances  (contract  number  SPM1C1-12-C0019),  each  for  90,000  helmets  (180,000 
helmets  total).  A  third  contract  solicitation  to  produce  a  lightweight  ACH  (8  percent 
weight  reduction),  has  yet  to  be  awarded.  The  bridge  contracts  follow  the  ballistic  testing 
requirements  of  the  legacy  test  protocol,  and  the  lightweight  ACH  solicitation  will  follow 
ballistic  testing  requirements  of  the  DOT&E  test  protocols.  Table  1  lists  ah  active  ACH 
procurements. 


Table  1.  ACH  Contracts  and  Solicitation 


Procurement 

Vendor 

Contract  Number 

Award  Date 

Ballistic  Testing 

ACH  Production 
Bridge  Contract 

Gentex 

Corporation 

SPM1C1-12-C-0016 

May  7,  2012 

Legacy  Test  Protocol 
(CO/PD-05-04) 

ACH  Production 
Bridge  Contract 

Mine  Safety 
Appliances2 

SPM1C1-12-C-0019 

June  14,  2012 

Legacy  Test  Protocol 
(CO/PD-05-04) 

Lightweight  ACH 

TBD 

TBD 

TBD 

Incorporated  DOT&E 
Protocol 

(AR/PD  10-02  Rev  A 
with  Change  4) 

ACH  RTP  Performance  Requirements 

The  ACH  is  designed  to  provide  ballistic  protection  from  fragments  as  well  as  9mm 
projectiles.  The  helmet  shell,  including  any  hardware  exposed  on  the  outside  of  the  shell, 
is  designed  to  be  resistant  to  a  9mm  Full  Metal  Jacketed  Round  Nose  (FMJ  RN}  Remington 
bullet  penetration  with  a  nominal  mass  of  124  grains.  The  9mm  FMJ  RN  ballistic  test 
is  an  industry  standard  adopted  from  the  National  Institute  of  Justice  (NIJ)  Standard 
0101.06,  “Ballistic  Resistance  of  Body  Armor."  It  was  also  derived  from  the  Operational 
Requirements  Document  (ORD)  for  the  Land  Warrior,  which  outlines  the  requirements 


2  The  division  of  Mine  Safety  Appliances  that  produced  ACHs  was  purchased  by  Revision  Military  Limited  in  June  2012. 
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and  operational  capability  needs  of  an  integrated  soldier  protection  and  equipment 
system.  The  ORD  states  that  the  greatest  threat  to  the  land  warrior  is  fragmentation  and 
the  second  greatest  threat  is  bullets.  For  the  ACH,  the  9mm  FMJ  RN  test  does  not  only 
represent  a  capability  to  be  resistant  against  bullets  but  also  larger  sized  fragments. 

The  ACH  is  not  designed  to  provide  ballistic  protection  from  threats  more  lethal  (for 
example,  higher  velocity,  or  larger  mass)  than  a  9mm  FMJ  RN.  Field  data  indicate  that  the 
ACH  performs  well  against  its  intended  threats,  but  is  penetrable  from  rifle  threats  that 
are  most  commonly  seen  in  theater.  A  new  product  called  the  Enhanced  Combat  Helmet 
(ECH)  is  currently  under  design  and  development  to  defeat  threats  more  lethal  than  a 
9mm  FMJ  RN. 

ACH  RTP  Ballistic  Testing 

According  to  contract  requirements,  the  ACH  is  required  to  undergo  FAT  and  LAT.  FAT 
is  conducted  on  a  specified  sample  size  and  determines  whether  the  proposed  product 
and  manufacturing  design  meets  testing  requirements  prior  to  proceeding  to  production. 
LAT  determines  whether  a  produced  lot  meets  testing  requirements  prior  to  delivery. 
Both  FAT  and  LAT  for  the  ACH  consists  of  non-ballistic  and  ballistic  test  and  inspection 
requirements.  This  report  focuses  on  the  RTP  ballistic  testing  requirements. 

ACH  RTP  ballistic  testing  for  the  helmet  consists  of  firing  9mm  projectiles  at  a  defined 
velocity  and  distance  from  the  helmet  to  demonstrate  a  required  level  of  penetration 
protection.  RTP  test  requirements  include  the  acceptable  number  of  complete  9mm  FMJ 
RN  penetrations3  for  each  sample  size. 

DOT&E  Role  in  Combat  Helmet  Testing 

In  2007,  both  the  House  Armed  Services  Committee  and  Senate  Armed  Services  Committee 
wrote  to  the  Secretary  of  Defense  expressing  their  desire  and  support  for  DOT&E 
involvement  in  body  armor  testing.  The  United  States  Congress  amended  Public  Law  110- 
417,  “Duncan  Hunter  National  Defense  Authorization  Act  for  Fiscal  Year  2009,"  to  expand 
the  Secretary  of  Defense's  authority  to  designate  programs  for  realistic  survivability 
testing.  The  Secretary  of  Defense  subsequently  authorized  DOT&E's  involvement  in  body 
armor  testing  and  updated  DoD  Directive  5141.02,  “Director  of  Operational  Test  and 
Evaluation,"  to  state  that  DOT&E  shall  prescribe  policies  and  procedures  for  the  conduct  of 


3  According  to  AR/PD  10-02  Rev  A  with  Change  4,  May  8,  2012,  "Purchase  Description  Helmet,  Advanced  Combat  (ACH)," 
complete  penetration  shall  be  defined  as  complete  perforation  of  the  shell  by  the  projectile  or  fragment  of  the  projectile 
as  evidenced  by  the  presence  of  that  projectile,  projectile  fragment,  or  spall  (fragments  of  the  test  helmet  being  impacted, 
excluding  fibrous  material,  paint,  and  epoxy  particles  emitted  from  the  helmet  surface)  in  the  clay,  or  by  a  hole  which 
passes  thru  the  shell. 
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Live  Fire  Test  and  Evaluation  (LFT&E).  LFT&E  includes  Personnel  Protective  Equipment 
(PPE)  testing  such  as  hard  body  armor,  soft  body  armor,  and  helmets. 

In  response  to  a  January  2009  DoDOIG  Report  No.  D-2009-047,  "DoD  Testing  Requirements 
for  Body  Armor"  DOT&E  published  the  hard  body  armor  FAT  protocol  on  April  27,  2010, 
and  the  LAT  protocol  on  July  2, 2010.  On  December  7, 2010,  DOT&E  published  the  Military 
Combat  Helmet  Standard  for  Ballistic  Testing  FAT  applicable  to  all  DoD  combat  helmet 
acquisition  programs,  including  the  ACH.  The  DOT&E  FAT  protocol  states  that  combat 
helmets  must  meet  a  set  of  statistically  based  FAT  to  qualify  a  design/manufacturing 
process  for  full-rate  production.  The  FAT  protocol  was  updated  on  September  2,  2011, 
to  include  both  aramid-based  helmets,  such  as  the  ACH  and  Lightweight  Helmet  (LWH), 
and  ultra-high  molecular  polyethylene-based  helmets,  such  as  the  ECH,  designed  to 
defeat  threats  more  lethal  than  a  9mm  FMJ  RN.  On  January  19,  2012,  DOT&E  released  the 
Military  Combat  Helmet  Standard  for  Ballistic  Lot  Acceptance  Testing  protocol,  detailing 
test  requirements  that  must  be  met  prior  to  accepting  a  production  lot  from  a  vendor. 
It  was  updated  on  May  4,  2012,  with  revisions  to  the  RTP  test  matrix  for  aramid-based 
helmets. 

FAT  RTP  Requirements 

The  FAT  RTP  legacy  requirements  used  by  ACH  bridge  contracts  were  established  in  the 
Contract  Purchase  Description  "CO/PD-05-04,”  October  30,  2007  (most  recently  updated 
on  March  24,  2009).  CO/PD-05-04  requires  a  sample  size  of  four  helmets  tested  under 
different  environmental  conditions  and  shot  five  times.  Under  CO/PD-05-04,  one  pen¬ 
etration  of  the  total  20  shots  will  result  in  a  failed  FAT.  The  sample  size  and  number  of 
allowable  penetrations  are  not  derived  from  a  known  statistical  model. 

In  contrast  to  the  legacy  FAT  RTP  requirement,  the  DOT&E  FAT  RTP  requirement  is  statis¬ 
tically  based.  The  FAT  RTP  requirement  increases  the  sample  size  to  48  helmets  at  5  shots 
per  helmet,  totaling  240  shots  based  on  an  established  "90/90  standard.”4  An  increase  in 
sample  size  increases  the  statistical  confidence  of  the  testing  result. 

The  Clopper-Pearson  method  is  used  to  calculate  the  allowable  number  of  penetrations 
out  of  a  given  sample  size.  When  applied  against  the  required  parameters  (90/90  and  240 
shots),  the  Clopper-Pearson  method  yields  17  allowable  penetrations.  A  more  detailed 
explanation  of  the  Clopper-Pearson  method  can  be  found  in  Appendix  C.  Table  2  shows  a 
comparison  of  the  legacy  FAT  RTP  and  DOT&E  FAT  RTP. 


4  The  commonly  termed  "90/90  standard"  for  First  Article  Testing  (FAT)  Resistance  to  Penetration  (RTP)  requirement  is  at 
least  90%  Probability  of  no  Penetration  (P(nP))  with  90%  Confidence  Level. 
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Table  2.  Comparison  of  FAT  RTP  Protocols 


9  mm  RTP 
shell 

Ambient 
70°  F 

Hot 

160°  F 

Cold 
-60°  F 

Immerse  in 
Seawater, 
then  test  at 
70°  F 

Totals 

#  Allowable 
Penetrations 
Accept/ 
Reject 

Legacy  test 
protocol 
sample  size 

5  shots 

5  shots 

5  shots 

5  shots 

20  shots 

0/1 

1  helmet 

1  helmet 

1  helmet 

1  helmet 

4  helmets 

DOT&E  test 
protocol 
sample  size 

60  shots 

60  shots 

60  shots 

60  shots 

240  shots 

17/18 

12  helmets 

12  helmets 

12  helmets 

12  helmets 

48  helmets 

LAT  RTP  Requirements 

The  legacy  LAT  RTP  requirement  used  by  ACH  bridge  contracts  were  also  established  in 
CO/PD-05-04.  CO/PD-05-04  defines  the  sample  size  per  lot  size  delivered  in  Table  3,  and 
specifies  that  one  9mm  penetration  will  result  in  a  failed  lot  regardless  of  the  lot  size. 


Table  3.  Legacy  LAT  RTP 


Lot  Size 

Sample  Size 

Accept 

Reject 

4-150 

5  shots 

1  helmet 

0 

1 

151-1,200 

5  shots 

1  helmet 

0 

1 

1,201-3,200 

10  shots 

2  helmets 

0 

1 

In  contrast  to  the  legacy  LAT,  the  DOT&E  LAT  RTP  requirement  is  4  percent  AQL.5  The 
sampling  plan  to  determine  the  accept/reject  criteria  is  based  on  the  American  National 
Standards  Institute  (ANSI)  American  Society  for  Quality  (ASQ)  Zl.4-2008  with  a  special  in¬ 
spection  level  S-3.  These  translate  to  the  sample  size  and  accept/reject  criteria  in  Table  4. 


Table  4.  DOT&E  LAT  RTP 


Lot  Size 

Sample  Size 

Accept 

Reject 

91-150 

25  shots 

5  helmets 

0 

1 

9  mm  RTP  '1  JfisSS 

shell 

40  shots 

8  helmet 

1 

2 

501-1,200 

65  shots 

13  helmets 

1 

2 

1,200+ 

65  shots 

13  helmets 

1 

2 

5  According  to  ANSI/ASQ  Zl.4-2008,  AQL  is  the  quality  level  that  is  the  worst  tolerable  process  average  when  a  continuing 
series  of  lots  is  submitted  for  acceptance  sampling. 
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Implementation  of  DOT&E  Helmet  Test  Protocols 

Because  the  DOT&E  FAT  and  LAT  protocols  are  not  intended  to  be  applied  against  already 
qualified  designs,  the  ACH  bridge  contracts  SPM1C1-12-C-0016  and  SPM1C1-12-C0019 
were  precluded  from  implementing  the  DOT&E  test  protocol  requirements.  However, 
PEO  Soldier  was  required  to  incorporate  the  DOT&E  test  protocols  in  the  lightweight 
ACH  specification.  The  incorporated  test  protocols  are  detailed  in  the  Contract  Purchase 
Description,  "AR/PD  10-02  Rev  A  with  Change  4,"  May  8,  2012. 


In  "AR/PD  10-02  Rev  A  with  Change  4,”  PEO  Soldier  established  a  "hybrid  FAT”  and  a  "hy¬ 
brid  LAT”  in  an  effort  to  combine  the  legacy  test  protocols  with  the  DOT&E  test  protocols. 
The  hybrid  FAT  requires  that  the  first  22  shots  yield  zero  penetrations  prior  to  conduct¬ 
ing  the  full  DOT&E  FAT  protocol.6  The  hybrid  LAT  requires  that  the  first  5  or  10  shots 
(depending  on  lot  size]  yield  zero  penetrations  prior  to  conducting  the  full  DOT&E  LAT 
protocol.  According  to  PEO  Soldier,  the  hybrid  test  strategy  ensures  that  the  pedigree  of 
the  past  helmet's  testing  requirements  continues  into  the  lightweight  ACH  procurement. 
Both  RTP  hybrid  FAT  RTP  and  LAT  RTP  summaries  are  depicted  in  Figure  2  and  3. 


Figure  2.  Hybrid  FAT  RTP 


Zero  (0) 
Penetrations 


22  Impacts 

(at  least  5  impacts 
in  each  condition) 

>  One  (1) 
Penetration 


Figure  3.  Hybrid  LAT  RTP 


Lot  Size  91-150: 

O  Penetration 
Lot  Size  151-500: 

1  Penetration 
Lot  Size  501-1200: 

1  Penetration 

Lot  Size  1201-3200: 

1  Penetration 


Lot  Size  91-150: 

1  Penetration 

Lot  Size  151-500: 

2  Penetration 

Lot  Size  501-1200: 

2  Penetration 

Lot  Size  1201-3200: 

2  Penetration 


The  Clopper-Pearson  method  for  binomial  confidence  intervals  requires  a  minimum  sample  size  of  22. 

-David  W.  Webb,  Army  Research  Laboratory,  A  Comparison  of  Various  Methods  Used  to  Determine  the  Sample  Size 
Requirements  for  Meeting  a  90/90  Reliability  Specification,  March  2011,  page  28 
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Notice  of  Concern 


We  issued  a  Notice  of  Concern  (NOC)  to  DOT&E  and  PEO  Soldier  on  September  12,  2012, 
stating  concerns  regarding  the  implementation  of  the  new  helmet  protocol  in  the  light¬ 
weight  ACH  solicitation.  We  recommended  that  PEO  Soldier,  DOT&E,  and  DLA  TS  work 
together  to  ensure  that  there  is  no  reduction  in  helmet  performance  for  the  new  light¬ 
weight  ACH.  DOT&E  released  a  coordinated  response  with  PEO  Soldier  and  DLA  TS  on 
October  12,  2012,  detailing  an  initial  set  of  plans  to  address  the  NOC. 

A  memorandum  signed  by  the  Deputy  Inspector  General  for  Policy  and  Oversight  was  is¬ 
sued  on  December  5,  2012,  to  document  concerns  with  the  helmet  protocols  and  actions 
that  DOT&E  and  PEO  Soldier  will  take  to  address  those  concerns.  The  memorandum  also 
concluded  that  the  NOC  was  resolved  by  DOT&E  and  PEO  Soldier’s  commitments  to  ad¬ 
dress  the  issues  in  light  of  their  October  12,  2012,  NOC  response.  A  copy  of  the  NOC  and 
the  resolution  memorandum  is  in  Appendix  D. 
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Finding  A 

Origin  of  Resistance  to  Penetration  Requirements 

The  DOT&E  test  protocol  for  the  ACH  adopts  a  statistically  principled  approach  and 
represents  an  improvement  from  the  legacy  test  protocol  with  regard  to  increased  sample 
size.  However,  future  protocol  revisions  necessitate  further  refinement  by  anchoring  the 
RTP  requirements  to  helmet  specific  empirical  data  such  as  manufacturing  capabilities 
and  test  performance. 


Discussion 

Hard  Body  Armor  Test  Protocol  Studies 

On  January  29,  2009,  the  DoD  OIG  issued  an  audit  report  on  DoD  Testing  Requirements 
for  Body  Armor  (Report  No.  D-2009-047,  "DoD  Testing  Requirements  for  Body  Armor") 
and  recommended  that  DOT&E  develop  a  test  operations  procedure  for  body  armor  in¬ 
serts  and  verify  that  the  procedure  is  implemented  DoD-wide.  In  response  to  the  DoD  OIG 
report,  DOT&E  and  PEO  Soldier  collaborated  with  the  Army  Test  and  Evaluation  Center 
(ATEC)  to  conduct  a  comprehensive  technical  assessment  of  the  hard  body  armor.  The 
technical  assessment  included  ballistic  performance  evaluation  of  the  Enhanced  Small 
Arms  Protective  Insert  and  the  X  Threat  Small  Arms  Protective  Insert  against  a  set  of 
defined  threats.  DOT&E  used  the  results  of  the  study  to  establish  the  body  armor  test 
protocols.  On  April  27,  2010,  DOT&E  issued  the  hard  body  armor  FAT  protocol,  defining 
FAT  RTP  90/90  standard  for  the  first  test  shot  and  a  70/90  for  the  second  test  shot. 

DOT&E  also  commissioned  the  National  Academies7  to  assess  the  methodologies  used 
by  the  U.S.  Army  for  body  armor  testing  and  assist  in  addressing  shortcomings  that  had 
been  reported  by  the  Government  Accountability  Office  and  the  DoD  OIG.  The  National 
Academies  released  the  results  of  their  assessment  in  three  phases.8  The  Phase  II  report 
addressed  DOT&E’s  statistically  based  body  armor  test  protocol  and  concluded  that,  "The 
new  DOT&E  [hard  body  armor]  protocol  meets  both  key  protocol  design  requirements;  it 
is  statistically  principled  and  it  provides  a  minimum  DoD-wide  body  armor  test  standard.” 


7  The  National  Academies  is  composed  of  the  National  Academy  of  Sciences,  National  Academy  of  Engineering,  Institute  of 
Medicine,  and  National  Research  Council.  These  organizations  are  private,  nonprofit  institutions  that  consist  of  elected 
members  with  distinguished  achievement  in  their  respective  fields.  The  National  Academies  is  known  for  providing 
objective  advice  on  science,  engineering,  and  health  matters.  Source:  http://www.nas.edu 

8  The  Phase  I  report  was  released  on  December  30,  2009,  Phase  II  on  April  22,  2010,  and  Phase  III  on  May  21,  2012. 
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The  National  Academies  recommended  that  DOT&E  implement  a  statistically  based  LAT 
that  incorporates  aspects  of  military  standard  MIL-STD-1916,  "DoD  Preferred  Methods 
for  Acceptance  of  Product."  Subsequently,  DOT&E  released  the  body  armor  LAT  protocol 
on  July  2, 2010  with  a  defined  LAT  RTP  requirement  of  4  percent  AQL  for  the  first  test  shot 
and  15  percent  AQL  for  the  second  shot.  The  National  Academies  did  not  recommend 
specific  RTP  requirements  for  LAT. 

The  National  Academies  also  recommended  that  standards  in  the  protocol  should  be  based 
on  empirical  evidence.  However,  DOT&E  stated  there  is  not  a  comparable  body  of  data 
that  characterizes  the  performance  of  the  ACH.  DOT&E  also  stated  in  a  memorandum  to 
Representative  Slaughter,  dated  July  13, 2012,  that  the  ACH  test  protocols  were  developed 
to  be  analogous  to  the  body  armor  test  protocols.  Although  DOT&E  established  a  helmet 
test  protocol  that  adopts  a  statistically  principled  approach  and  represents  an  improve¬ 
ment  from  the  legacy  test  protocol,  the  protocol’s  RTP  requirements  are  not  anchored  to 
helmet  specific  empirical  data  such  as  manufacturing  capabilities  and  test  performance. 

Common  RTP  Requirements  Across  PPEs 

It  was  DOT&E’s  intent  to  publish  common  RTP  requirements  across  all  PPEs.  According 
to  DOT&E,  one  of  the  driving  factors  for  a  common  standard  is  public  perception.  If  each 
individual  type  of  PPE  had  specific  standards,  questions  may  arise  as  to  why  one  PPE  had 
a  lower  standard  than  another.  DOT&E  also  stated  that  it  would  be  potentially  difficult 
and  unmanageable  to  have  individual  protocols  and  designs  for  each  piece  of  PPE.  Fur¬ 
thermore,  DOT&E  stated  that  a  common  standard  would  provide  some  level  of  assurance 
that  all  fielded  commodities  have  a  minimum  level  of  performance. 

DOT&E  referenced  the  DoD  OIG  audit  report  on  body  armor  and  a  series  of  studies  con¬ 
ducted  by  the  National  Academies  from  2009  to  2012  on  body  armor.  The  DoD  OIG  re¬ 
port  states  that  DoD  does  not  have  standardized  ballistic  testing  criteria  for  body  armor 
inserts.  Additionally,  the  National  Academies  Phase  III  report  on  body  armor  stated  that 
an  encompassing  standard  for  testing  would  include  alignment  of  body  armor  and  helmet 
testing  procedures  and  processes.  However,  neither  of  the  two  referenced  reports  pre¬ 
scribe  common  parameters  such  as  the  FAT  RTP  90/90  standard  or  LAT  RTP  of  4  percent 
AQL  across  all  PPEs. 

DOT&E  Assessment  Discussions 

During  the  course  of  our  assessment,  DOT&E  and  PEO  Soldier  committed  to  fully 
characterizing  the  performance  of  the  ACH  in  response  to  our  NOC  detailed  in  Appendix 
D.  DOT&E  also  indicated  that  it  will  also  work  with  the  Marine  Corps  to  characterize 
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the  performance  of  ECH.  DOT&E  will  provide  the  details  of  the  plans  to  characterize  the 
helmets  to  the  DoD  OIG  when  the  respective  plans  are  approved.  DOT&E  added  that  as 
new  designs/technology  become  available,  it  will  consider  the  characterization  of  those 
designs  during  developmental  testing  to  assess  whether  any  changes  to  the  test  protocol 
may  be  appropriate. 

DOT&E  acknowledged  that  the  performance  metrics  incorporated  in  test  standards  used 
for  PPE  should  be  empirical,  based  on  the  data  generated  by  characterization  studies  con¬ 
ducted  on  the  individual  elements  of  PPE  using  the  appropriate  threats.  DOT&E  stated 
that  deviating  from  a  90/90  standard  could  be  appropriate,  based  on  the  performance 
revealed  by  these  characterization  studies  against  the  specified  threats.  DOT&E  consid¬ 
ers  the  established  test  standards  for  hard  body  armor  plates  and  helmets  that  reflect  a 
common  metric  to  have  elements  that  are  generally  applicable  to  all  PPE.  Specifically,  the 
standards  for  PPE  testing  must  be  as  statistically  principled  and  rigorous. 

DOT&E  also  commissioned  the  National  Academies  on  August  15,  2012,  to  determine  the 
adequacy  of  their  military  ballistic  helmet  testing.  DOT&E  estimates  that  the  National 
Academies  will  complete  its  study  by  December  2013. 

Recommendation 

We  recommend  that  DOT&E  and  PEO  Soldier  fully  characterize  the  performance  of  all 
helmet  designs  included  in  the  combat  helmettest  protocols.  Performance  characterization 
should  consider  threat,  historical  test  data,  prototype  test  data,  and  manufacturing 
capabilities.  Based  on  helmet  performance  characterizations,  DOT&E  and  PEO  Soldier 
should  determine  if  modification  to  the  FAT  and  LAT  protocols  are  appropriate. 

DOT&E  and  PEO  Soldier  Comments 

Characterization  on  fielded  ACH  will  rely  on  historical  test  data  since  the  Army  does  not 
intend  to  buy  additional  ACH  designs.  We  will  conduct  additional  testing  to  characterize 
helmets  not  currently  fielded,  such  as  the  Lighweight  ACH,  the  Enhanced  Combat  Helmet 
(ECH),  the  Soldier  Protective  System  Integrated  Head  Protection  System  (SPS  IHPS).  This 
characterization  will  be  based  on  a  risk  assessment  that  identifies  and  documents  the 
testing  that  provides  the  most  important  data,  since  it  is  unrealistic  to  scope  a  charac¬ 
terization  effort  that  includes  every  desirable  test.  Based  on  the  characterization  data, 
DOT&E  will  determine  whether  FAT  and  LAT  protocols  should  be  modified. 
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DoD  OIG  Response 

DOT&E  and  PEO  Soldier’s  comments  were  fully  responsive.  In  addition  to  the  response, 
we  recommend  that  the  characterization  include  the  Industry’s  state  of  the  art  with  re¬ 
gard  to  controlling  manufacturing  parameters  of  the  design.  No  additional  comments 
from  DOT&E  and  PEO  Soldier  are  required. 
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Finding  B 

Acceptable  Quality  Level  Based  on  Safety  Criticality 

The  DOT&E  LAT  protocol  is  an  improvement  from  the  legacy  LAT  and  adopts  a  widely 
established  and  industrially  accepted  American  National  Standards  Institute  (ANSI  Z1.4- 
2008,  Sampling  Procedures  and  Tables  for  Inspection  by  Attributes).  In  selecting  the  LAT 
RTP  requirement  of  4  percent  AQL,  DOT&E  considered  the  government  risk  of  accepting 
underperforming  helmets,  manufacturer  risk  of  failing  LAT  with  acceptable  helmets,  and 
historical  LAT  data.9  However,  DOT&E  did  not  consider  selecting  an  AQL  that  was  based 
on  the  safety  criticality  of  the  helmet. 


Discussion 

LAT  Sampling  Plan 

The  DOT&E  LAT  RTP  sampling  plan  was  derived  from  a  widely  established  and  industri¬ 
ally  accepted  ANSI  Z1.4  -2008.  Sampling  plans  are  particularly  applicable  to  destructive 
testing10  such  as  the  9mm  RTP  test  performed  during  LAT,  when  not  all  helmets  in  a  given 
lot  can  be  tested.  The  DOT&E  LAT  RTP  sampling  plan  is  derived  from  inspection  level 
S-3.* 11  ANSI  Zl.4-2008  also  provides  reference  to  operating  characteristic  curves  to  deter¬ 
mine  the  government  risk  and  manufacturer  risk  at  a  given  AQL. 

AQL  Selection  Based  on  Criticality 

Although  the  use  of  a  widely  established  standard  represents  an  improvement  from 
the  legacy  LAX  DOT&E  did  not  consider  selecting  an  AQL  that  was  based  on  the  safety 
criticality  of  the  helmet.  The  Land  Warrior  Operational  Requirements  Document  (ORD) 
states  that  the  primary  function  of  the  helmet  is  to  protect  the  Land  Warrior's  head  from 
fragmentation  and  bullets  and  therefore,  a  helmet  is  considered  a  safety  critical  item  and 
resistance  to  penetration  is  considered  a  critical  characteristic. 

If  a  sampling  plan  is  not  specified  in  the  contract,  the  Defense  Contract  Management 
Agency  (DCMA)12  will  implement  DCMA  Instruction  226-3,  “Critical  Safety  Items 


9  Government  risk  is  the  probability  of  accepting  underperforming  helmets  while  manufacturer  risk  is  the  probability  of 
failing  LAT  with  acceptable  helmets 

10  Destructive  testing  is  a  test  that  puts  the  sample  object  under  certain  circumstances  until  it  actually  fails. 

-H.F.  Walker,  A.K.  Elshennawy,  B.C.  Gupta,  M.  McShane-Vaughn,  The  Certified  Quality  Inspection  Handbook ,  Second  Edition, 
(Milwaukee:  American  Society  for  Quality,  2013)  page  186. 

11  Per  ANSI  Z1.4,  the  inspection  level  determines  the  relationship  between  the  lot  or  batch  size  and  the  sample  size. 

12  The  DCMA  was  established  to  perform  contract  administration  for  the  Department  of  Defense.  DCMA  is  expected  to 
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(CSIs)"  and  Instruction  226-4,  "Product  Examination  -  Quality  Assurance  (QA) DCMA 
Instruction  226-3  and  Instruction  226-14  both  state  that  non-aviation  CSI  includes 
personal  protective  devices  and  that  0.4  percent  AQL  is  applied  for  non-aviation  CSIs.  In 
addition  to  the  DCMA  requirement  of  0.4  percent  AQL,  ACH  manufacturers  indicated  that 
they  recommend,  and  are  currently  working  to  a  0.4  percent  AQL.  Lastly,  The  Certified 
Quality  Process  Analyst  Handbook  published  by  the  American  Society  for  Quality13  states 
that  in  practice,  critical  characteristics  are  commonly  inspected  to  an  AQL  of  0.4  to  0.65 
percent  if  not  100  percent  inspected. 

In  summary,  the  DCMA  instruction,  manufacturer  capabilities,  and  industry  standards 
warrant  tighter  constraints  on  components  or  assemblies  used  in  safety  critical  applica¬ 
tions,  thus  setting  the  AQL  at  a  lower  level  will,  over  a  continuing  series  of  lots,  provide 
the  worst  tolerable  process  average  appropriate  for  a  safety  critical  item. 

DOT&E  Assessment  Discussion 

Per  DOT&E,  the  new  LAT  protocol  increases  the  sample  size  used  for  LAT  and,  in  doing 
so,  decreases  the  risks  to  the  government  of  accepting  underperforming  helmets  because 
helmet  lots  under  the  previous  protocol  had  a  higher  chance  of  passing  LAT.  DOT&E  also 
considered  the  risk  to  the  manufacturer  of  failing  LAT  if  the  manufacturer  consistently 
produces  helmets  with  acceptable  performance.  Per  DOT&E,  the  new  and  old  LAT  proto¬ 
cols  have  similar  manufacturer’s  risk  for  a  helmet.  By  decreasing  government  risk  while 
maintaining  similar  manufacturer’s  risk,  DOT&E  stated  that  the  LAT  protocol  has  clearly 
improved  upon  the  previous  LAT.  DOT&E  has  provided  the  LAT  protocol  to  the  National 
Research  Council  (NRC)  committee  on  helmet  testing  and  asked  the  committee  to  evalu¬ 
ate  the  construction  of  the  current  LAT  protocol  and  to  provide  its  views  on  how  best  to 
relate  AQL  to  FAT  performance. 

Recommendation 

We  recommend  that  in  addition  to  considerations  made  to  Government  risk,  manufactur¬ 
er  risk,  and  historical  LAT  data,  DOT&E  should  also  consider  the  safety  criticality  of  the 
helmet  in  establishing  the  AQL  requirement  for  LAT  RTP  in  future  protocol  revisions. 


operate  in  an  independent,  consistent,  transparent,  and  collaborative  manner  while  performing  a  wide  variety  of  contract 
oversight  functions.  For  the  ACH  program,  DCMA  is  required  to  prepare  surveillance  strategies  for  all  ACH  contracts  to 
include  product  audits  and  inspections.  DCMA  provide  oversight  at  manufacturing  facilities  and  testing  facilities.  However, 
they  do  not  conduct  in-process  inspections  and  will  no  longer  need  to  provide  ballistic  test  surveillance  when  ACH  ballistic 
testing  begins  at  ATEC. 

13  E.H.  Christensen,  K.M.  Coombes-Betz,  M.S.  Stein,  The  Certified  Quality  Process  Analyst  Handbook,  (Milwaukee:  American 
Society  for  Quality,  2007) 


14  |  DODIG-2013-079 


Finding  B 


DOT&E  and  PEO  Soldier  Comments 

DOT&E  will  consider  helmet  safety  criticality  as  an  additional  consideration  in  estab¬ 
lishing  the  AQL  requirement  for  LAT  RTP  during  the  next  protocol  revision.  The  next 
protocol  revision  will  consider  recommendations  from  the  on-going  National  Academies 
review  of  helmet  test  protocols.  This  study  should  be  complete  by  December  2013. 

DoD  OIG  Response 

DOT&E  and  PEO  Soldier’s  comments  were  fully  responsive,  and  no  additional  comments 
are  required. 
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Coordination  of  Helmet  Test  Protocols 

In  accordance  with  authorizing  statutes,  DOT&E  has  the  authority  to  establish  test 
standards  for  personnel  protective  equipment  such  as  the  ACH.  However,  despite  the 
significance  and  broad  impact  of  these  protocols,  DOT&E  did  not  explicitly  consult  with 
heads  of  the  Military  Departments  to  provide  them  an  opportunity  to  comment  on  new 
or  changed  test  protocols  and  did  not  adequately  document  the  adjudication  of  inputs 
provided  by  program  offices  and  subject  matter  experts  in  the  staffing  process.  The 
program  office  also  did  not  solicit  comments  on  the  helmet  test  protocols  with  the  helmet 
vendors  and  DCMA. 


Discussion 

DoD  Directive  5141.02,  "Director  of  Operational  Test  and  Evaluation  (DOT&E),"  states 
that  DOT&E  shall  prescribe  policies  and  procedures  for  the  conduct  of  Live  Fire  Test 
and  Evaluation  (LFT&E)  and  delegate  authority  to  approve  LFT&E  strategies.  LFT&E 
strategies  include  PPE  testing  such  as  hard  body  armor,  soft  body  armor,  and  helmets. 

When  DOT&E  was  developing  the  body  armor  test  protocol,  the  National  Academies 
Phase  II  report  recommended  that  DOT&E  conduct  due  diligence  before  formally 
adapting  a  statistically  based  body  armor  protocol.  The  National  Academies  Phase  III 
report  recommended  that: 

DOT&E  should  provide  briefings  to  and  receive  feedback  from  all 
stakeholders  in  DoD  (military  service  Program  Executive  Officers, 
testers,  users)  and  non-DoD  organizations  (National  Institute  of 
Justice,  National  Institute  of  Standards  and  Technology,  certified 
private  testing  laboratories,  vendors)  concerning  the  statistical¬ 
ly  based  protocol.  DOT&E  should  act  on  feedback  from  the  com¬ 
munity  to  improve  the  proposed  protocol  as  necessary,  to  ensure 
that  testing  terms  and  concepts  make  sense  to  a  nontechnical 
audience,  and  it  should  promote  the  use  of  the  statistically  based 
protocols  in  future  national  standards  for  body  armor  testing,  as 
appropriate. 
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DoD  Leadership  Involvement 

DOT&E  stated  that  the  helmet  test  standards  are  policies  informed  by  a  technical 
understanding  of  both  the  testing  process  and  the  capabilities  of  the  item  under  test.  They 
also  provided  drafts  of  the  combat  helmet  FAT  and  LAT  protocols  to  various  stakeholders 
for  comments.  DOT&E  sent  the  FAT  protocol  draft  on  August  17,  2010,  and  it  underwent 
two  additional  review  iterations.14  DOT&E  also  sent  the  LAT  protocol  draft  on  February 
25,  2011.15  Subject  matter  experts  within  these  organizations  provided  comments  on 
the  protocol  drafts  in  areas  such  as  test  range  set  up,  sample  size,  and  ballistic  impact 
locations  and  measurements. 

Although  we  found  records  of  subject  matter  expert  involvement  (including  PEO  Soldier), 
we  found  no  coordinated  input  from  DoD  leadership,  including  heads  of  the  Military  De¬ 
partments.  There  was  no  objective  evidence  that  DoD  leadership  and  heads  of  the  Mili¬ 
tary  Departments  were  given  the  opportunity  to  provide  input  to  policies  that  establish 
an  acceptable  level  of  risk  (that  is,  acceptable  number  of  ballistic  penetrations)  to  the 
ACH. 

Documenting  Inputs 

PEO  Soldier  submitted  nine  nonconcurrences  (four  critical  and  five  substantive)  to 
DOT&E  after  reviewing  the  first  draft  of  the  combat  helmet  FAT  protocol.  PEO  Soldier  was 
concerned  that  a  statistically  based  protocol  that  allows  any  percentage  of  the  helmets 
to  be  penetrated  on  any  shot  during  FAT  could  result  in  unintended  consequences.  PEO 
Soldier  further  recommended  that  the  number  of  allowed  penetrations  be  vetted  before 
mandating  the  new  protocol.  DOT&E  responded  to  these  nonconcurrences  by  stating  that 
"discussion  will  occur."  We  found  no  evidence  that  DOT&E  incorporated  PEO  Soldier’s 
input  and/or  held  technical  discussions  with  PEO  Soldier  personnel  on  its  nonconcur¬ 
rences.  DOT&E  also  stated  that  there  was  no  further  action  on  these  nonconcurrences 
and  provided  no  official  response  to  the  nonconcurrences.16 


14  Recipients  of  the  FAT  protocol  drafts  were  departments  within  the  Marine  Corp  Systems  Command  (MCSC),  Naval  Sea 
Systems  Command  (NAVSEA),  Army  Research  Laboratory  Survivability/Lethality  Analysis  Directorate  (ARL/SLAD),  PEO 
Soldier  Program  Manager  Soldier  Protective  Equipment  (PM  SPE),  Deputy  Under  Secretary  of  the  Army  Test  and  Evaluation 
Office  (DUSA-TE),  DLA  Troop  Support  Clothing  and  Textile,  ATEC,  U.S.  Special  Operations  Command  (USSOCOM)  and 
Institute  for  Defense  Analysis  (IDA). 

15  Recipients  of  the  LAT  protocol  drafts  were  U.S.  Army  Acquisition  Support  Center  (USAASC),  U.S.  Marine  Corps  (USMC), 
Developmental  Testbed  Center  (DTC),  PM  SPE,  DLA,  U.S.  Army  Training  and  Doctrine  Command  (USATRADOC),  USSOCOM, 
U.S.  Army  Materiel  Command  (USA  AMC),  National  Institute  of  Standards  and  Technology  (NIST),  IDA,  Federal  Bureau  of 
Investigation  (FBI)  Academy,  U.S.  Air  Force  Acquisition  (USAF  SAF/AQRE),  ARL/SLAD,  ATEC,  DUSA-TE,  and  Office  of  the  Chief 
of  Naval  operations  (OPNAV). 

16  During  the  course  of  our  assessment,  PEO  Soldier  stated  that  they  were  in  agreement  with  the  90/90  standard  on  the  basis 
that  the  legacy  FAT  RTP  (0  penetrations  out  of  20  shots)  equates  to  89/90  when  analyzed  against  the  Clopper-Pearson 
method. 
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We  also  reviewed  documents  to  determine  the  origin  of  the  of  the  FAT  RTP  90/90 
standard  and  LAT  RTP  of  4  percent  AQL,  including  reports  and  studies  conducted  by 
both  the  National  Academies  and  ATEC  on  the  hard  body  armor  detailed  in  Finding  A. 
DOT&E  stated  that  in  addition  to  the  influence  of  the  National  Academies  and  ATEC 
studies  on  hard  body  armor,  the  RTP  FAT  and  LAT  requirements  were  recommended  by 
NRC  statisticians.  DOT&E  further  stated  that  the  National  Institute  of  Justice  (NIJ)  also 
expressed  that  a  90/90  standard  was  high  performing  and  a  good  protocol  for  any  PPE. 
However,  there  were  no  documented  records  of  the  recommendations  from  the  NIJ  or 
the  NRC,  specifically  for  the  helmet  test  protocols.  DOT&E  stated  that  recommendations 
were  mostly  informal  and  undocumented. 

Coordination  with  ACH  Vendors 

Although  the  National  Academies  made  the  recommendation  to  receive  vendor  feedback 
for  developing  body  armor  test  protocol,  the  same  due  diligence  of  receiving  vendor 
feedback  still  applies  to  developing  the  helmet  test  protocol.  However,  vendors  were 
not  asked  to  review  and  provide  input  to  the  helmet  protocols.  Vendors  have  years  of 
experience  in  design  and  manufacturing  of  helmets.  Significant  investment  in  research 
and  development  for  increased  performance,  weight  reduction,  and  cost  saving  have  been 
performed  by  vendors,  acquiring  a  depth  of  knowledge  that  could  have  been  useful  in 
developing  the  helmet  protocols. 

Another  recommendation  from  the  Phase  III  report  states  that  any  revision  to  the  proto¬ 
col  should  be  evaluated  to  ensure  that  there  is  no  detrimental  impact  on  the  quality  and 
performance  of  an  already  successful  product.  The  Phase  III  report  specifically  states: 

In  particular,  because  manufacturers  have  strong  incentives  to 
build  armor  that  has  a  high  chance  of  passing  FAT  and  LAT,  there  is 
some  chance  that  the  change  in  test  protocol  could  have  unintend¬ 
ed  impacts  on  body  armor  design  and/or  performance.  Given  the 
success  of  the  current  body  armor  in  the  field,  changes  in  testing 
protocols  should  be  made  with  deliberate  caution  to  ensure  that 
plate  performance  is  maintained  (or  improved)  while  also  ensur¬ 
ing  that  the  best  science  is  brought  to  bear  on  testing  body  armor. 

Similar  to  the  body  armor,  the  currently  fielded  ACHs  have  also  performed  well  against 
their  intended  threats.  ACH  vendors  also  disclosed  that  they  are  capable  of  performing 
well  above  the  RTP  test  protocol  requirements. 
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Role  of  the  Program  Manager 

The  DOT&E  helmet  test  protocols  do  not  explicitly  allow  program  managers  to  modify 
the  stringency  of  the  helmet  test  protocols.  Program  managers  may  need  protocol 
modifications  as  they  manage  programmatic  risks  associated  with  the  use  of  new 
manufacturing  processes,  materials,  or  design.  DOT&E  has  expertise  in  testing  but  lacks 
the  expertise  in  ACH  design  and  hardware. 

Program  managers’  limited  authority  over  testing  protocols  of  their  products  is  in  conflict 
with  the  requirements  of  DoDI  5000.02,  "Operation  of  the  Defense  Acquisition  System." 
DoD  Instruction  (I)  5000.02  defines  the  authority  of  the  program  manager  as  covering 
all  design,  manufacturing,  test,  and  all  other  program  life  cycle  processes.  In  addition, 
the  testing  protocols  can  limit  the  program  manager’s  ability  to  use  a  balanced  approach 
to  manage  programmatic  risks  such  as  the  use  of  new  manufacturing  processes  and  the 
use  of  new  materials  or  alternate  designs.  These  changes  may  necessitate  changes  to  the 
testing  protocols,  which  the  program  manager  does  not  control. 

DOT&E  Assessment  Discussion 

DOT&E  committed  to  provide  proposed  protocols  to  the  senior  leadership  of  the  affected 
Services  for  comment.  The  Director,  DOT&E,  will  consider  and  resolve  comments  received. 

DOT&E  stated  that  they  do  not  work  directly  with  vendors;  this  is  the  purview  of  program 
offices.  However,  DOT&E  has  and  will  continue  to  encourage  program  offices  to  provide 
vendors  the  opportunity  to  submit  through  program  offices  any  comments  on  proposed 
protocols. 

Currently,  DOT&E  allows  protocols  to  be  changed  and  made  more  stringent,  provided 
that  program  managers  submit  a  well-justified  request  for  approval.  DOT&E  will  state 
that  changes  are  permitted  in  future  protocol  updates,  but  DOT&E  retains  the  authority 
to  approve  such  changes. 

Recommendations 

C.l.  Due  to  the  significance  and  broad  impact  of  these  protocols,  we  recommend  that 
DOT&E  ensure  that  affected  organizations,  including  heads  of  the  Military  Departments, 
are  consulted  in  developing  the  protocols  and  have  an  opportunity  to  provide  input 
on  new  or  changed  test  protocols.  DOT&E  should  also  ensure  that  inputs  received  are 
documented  and  adjudicated. 
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C.2.  We  recommend  that  PEO  Soldier  solicit  input  from  helmet  vendors  and  DCMA  on 
new  or  changed  test  protocols.  PEO  Soldier  should  also  ensure  that  inputs  received  are 
documented  and  adjudicated. 

C.3.  We  recommend  that  DOT&E  include  an  explicit  statement  in  future  protocol  revisions 
that  allows  program  managers  to  modify  the  test  protocols,  provided  that  program 
managers  submit  a  well-justified  request  for  approval. 

DOT&E  and  PEO  Soldier  Comments 

DOT&E  will  ensure  affected  organizations,  including  heads  of  Military  Departments,  are 
consulted  in  developing  new  or  changed  protocols.  Responses  will  be  adjudicated  by  the 
DOT&E  and  documented.  A  request  to  both  the  Secretary  of  the  Army  and  Secretary  of 
the  Navy  to  review  the  current  protocol  resulted  in  no  recommendations  for  changes  to 
the  protocol  by  both  Departments. 

PEO  Soldier  will  solicit  feedback  from  our  industry  partners  as  well  as  DCMA  prior  to  the 
implementation  of  any  new  testing  protocol.  Industry  feedback  together  with  feedback 
from  many  various  engineers  working  for  PEO  Soldier  will  be  reviewed  and  adjudicated. 
All  adjudicated  feedback  will  be  provided  to  DOT&E  for  review  and  may  warrant  the  con¬ 
tinued  modification  of  existing  and  future  rest  protocols  and  standards. 

The  next  protocol  revision  will  include  an  explicit  statement  that  permits  program  man¬ 
agers  to  increase  the  test  protocol  stringency  following  approval  by  DOT&E  of  a  well-jus¬ 
tified  request  to  do  so. 

DoD  OIG  Response 

DOT&E  and  PEO  Soldier’s  comments  were  fully  responsive,  and  no  additional  comments 
are  required. 
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Evaluation  of  Aggregate  Test  Results 

The  FAT  RTP  acceptance  was  based  on  an  aggregate  of  all  test  outcomes  under  varying 
conditions  to  achieve  90/90,  or  17  penetrations  out  of  240  shots.  This  could  result  in 
passing  FAT,  despite  test  results  showing  clusters  of  failures  of  a  unique  helmet  size  or  in 
a  particular  test  environment. 


Discussion 

The  helmet  test  protocol  aggregates  the  FAT  RTP  test  results  across  environmental 
conditions,  helmet  sizes,  and  shot  locations  and  applies  the  Clopper-Pearson  method. 
The  Clopper-Pearson  method  employs  binomial  distribution  in  the  calculation  of 
bounds  of  the  confidence  interval.  In  general,  the  sum  of  binomial  distributions  do  not 
yield  a  binomial  distribution,  unless  the  failure  rates  are  the  same  for  each  individual 
environmental  condition  and  helmet  size.  Because  the  failure  rates  are  unknown  for  each 
individual  environmental  condition,  we  cannot  assume  that  the  aggregated  test  result 
represents  a  binomial  distribution  and  therefore  the  test  results  cannot  validate  that  the 
90/90  standard  has  been  met. 

Implications  of  Aggregating  Test  Results 

In  a  July  13, 2012  letter,  DOT&E  provided  a  response  to  Representative  Slaughter’s  inquiry 
into  the  FAT  test  protocol.  The  DOT&E  response  stated  that, 

...  the  new  protocol  samples  a  large  enough  number  of  helmets  for 
testing  to  demonstrate  statistically  significant  differences  in  per¬ 
formance  due  to  environmental  conditions  (such  as  hot  and  cold 
temperatures),  among  helmet  sizes,  and  among  different  vendors’ 
designs  ... 

A  review  of  the  DOT&E  test  data  analysis  methods  identified  that  the  results  are  aggregated 
across  different  environmental  conditions,  helmet  sizes,  and  shot  locations.  Analyzing 
aggregated  test  data  when  determining  whether  the  results  pass  or  fail  RTP  FAT  masks 
statistically  significant  differences  in  performance  due  to  environmental  conditions, 
helmet  sizes,  and  different  vendor  designs. 
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The  following  scenarios  illustrate  the  implications  of  aggregating  test  results  using  envi¬ 
ronmental  conditions  as  the  only  variable,  without  introducing  the  other  variables  such 
as  helmet  sizes,  shot  locations,  and  vendor  designs. 

Scenario  1  demonstrates  that  individually,  all  of  these  conditions  fail  to  meet  a  90/90 
standard.  But  if  the  individual  results  are  combined  and  the  Clopper-Pearson  method  is 
applied,  the  combined  sample  results  meet  the  standard. 


Table  5.  Scenario  1 


Conditions 

Number  of  Shots 

Number  of  Failures 

LCL  of  P(nP) 

Ambient 

60 

5 

85% 

Hot 

60 

4 

87% 

Cold 

60 

4 

87% 

Seawater 

60 

4 

87% 

Total 

240 

17 

90% 

Scenario  2  demonstrates  the  possibility  of  17  failures  during  ambient  conditions.  The  test 
results  in  ambient  conditions  fail  to  meet  the  standard,  while  the  combined  result  of  17 
failures  out  of  240  shot  appears  to  meet  the  standard. 


Table  6.  Scenario  2 


Conditions 

Number  of  Shots 

Number  of  Failures 

LCL  of  P(nP) 

Ambient 

60 

17 

63% 

Hot 

60 

0 

96% 

Cold 

60 

0 

96% 

Seawater 

60 

0 

96% 

Total 

240 

17 

90% 

In  both  the  scenarios,  the  combined  result  of  17  failures  out  of  240  shots  meet  the  90/90 
standard.  This  could  result  in  passing  FAT,  despite  test  results  showing  failures  in  a 
particular  test  environment. 

DOT&E  Assessment  Discussion 

DOT&E  was  aware  of  these  implications,  and  committed  to  perform  additional  statistical 
analysis  beyond  that  specified  in  the  protocol.  With  respect  to  aggregation  of  helmet  test 
data,  DOT&E  will  analyze  whether  significant  clusters  of  inadequate  performance  exist. 
DOT&E  will  use  either  logistic  regression  analysis  or  two-sample  tests  of  proportions  to 
analyze  the  probability  of  no  penetration  for  significant  clusters.  Significant  results  would 
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trigger  additional  testing  to  determine  whether  helmet  protocol  requirements  were  met 
by  condition.  DOT&E  will  issue  a  memorandum  describing  the  technical  details  of  this 
analysis  and  will  subsequently  incorporate  this  approach  in  a  published  update  to  the 
existing  test  protocol. 

DOT&E  and  PEO  Soldier  will  adopt  this  approach  with  the  current  lightweight  ACH 
solicitation,  but  not  as  a  published  change  to  the  test  protocol  specified  in  the  current 
solicitation.  Although  the  solicitation  will  not  be  changed,  DOT&E  will  analyze  the  test 
results  obtained  under  the  current  solicitation  to  see  if  significant  differences  (based  on 
environment,  helmet  size,  or  shot  location)  exist,  and  if  so,  will  conduct  additional  testing 
to  determine  the  FAT  RPT  criteria  under  the  test  condition  of  interest.  If  the  results 
demonstrate  poor  performance,  DOT&E  will  not  field  the  lightweight  ACH  design. 

Recommendations 

D.l.  We  recommend  that  DOT&E  and  PEO  Soldier  describe  the  method  of  identifying 
and  addressing  statistically  significant  differences  in  performance  due  to  environmental 
conditions,  helmet  sizes,  shot  locations,  and  different  vendor  designs  for  all  FAT  results 
under  the  DOT&E  helmet  test  protocol. 

D.2.  We  recommend  that  PEO  Soldier  and  DLA  consider  the  contractual  implications  of 
not  fielding  an  ACH  design  that  passes  FAT  yet  shows  significant  clusters  of  inadequate 
performance. 

DOT&E  and  PEO  Soldier  Comments 

DOT&E  has  developed  procedures  for  identifying  and  addressing  statistically  significant 
differences  in  performance.  DOT&E  has  asked  the  National  Academies,  as  part  of  its  re¬ 
view  of  the  helmet  test  protocols,  to  comment  on  the  adequacy  of  these  procedures  to 
identify  significant  differences  in  performance.  DOT&E  will  incorporate  these  proce¬ 
dures  (as  revised  based  on  feedback  from  the  National  Academies)  in  the  next  revision 
of  the  protocol.  DOT&E  has  provided  a  copy  of  these  procedures  to  the  DoD  OIG  staff  for 
information. 

DoD  OIG  Response 

DOT&E  and  PEO  Soldier’s  comments  were  fully  responsive.  We  reviewed  DOT&E’s  de¬ 
tailed  procedures  for  identifying  and  addressing  statistically  significant  differences  in 
performance  and  have  additional  points  for  consideration.  However,  no  additional  com¬ 
ments  from  DOT&E  and  PEO  Soldier  are  required. 
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The  DOT&E  strategy  to  identify  significant  differences  and  trends  in  performance  is  a 
statistical  approach  to  determine  or  trigger  whether  additional  testing  is  required.  In 
addition  to  this  approach,  we  recommend  that  PEO  Soldier  designate  the  triggering  of  a 
cluster  analysis  as  a  nonconformance,  requiring  further  engineering  evaluation  and  dis¬ 
position.  Failures  under  certain  conditions  can  be  attributed  to  design,  manufacturing, 
material  weaknesses,  or  test  equipment/test  conduct  variability.  The  root  cause  of  these 
failures  and  subsequent  mitigation  plans  should  be  a  product  of  an  established  failure 
analysis  and  corrective  action  process  that  involve  the  vendor,  PEO  Soldier’s  Quality  As¬ 
surance  representatives,  and  DCMA. 
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Scope  and  Methodology 

We  conducted  this  assessment  from  July  2012  through  February  2013  in  accordance 
with  the  Council  of  the  Inspectors  General  on  Integrity  and  Efficiency,  "Quality  Standards 
for  Inspection  and  Evaluation."  Those  standards  require  that  we  plan  and  perform  the 
assessment  by  obtaining  sufficient  and  appropriate  evidence  to  provide  a  reasonable 
basis  for  our  conclusions.  We  believe  that  the  evidence  obtained  provides  a  reasonable 
basis  for  our  conclusions. 

To  assess  the  basis  for  the  RTP  requirements  and  determine  whether  the  RTP  require¬ 
ments  are  appropriate  for  the  ACH,  the  DoD  OIG  team  assessed  three  different  areas:  ACH 
test  protocols  and  standards,  test  processes,  and  production  and  procurement  processes. 

We  conducted  interviews  with  DOT&E,  PEO  Soldier,  DLA,  DCMA,  ATEC,  and  ACH  vendors. 
We  wrote  summaries  after  each  meeting  that  were  signed  and  agreed  upon  by  attending 
organizations. 

We  conducted  assessments  at  DOT&E  and  PEO  Soldier  to  determine  PEO  Soldier’s 
relationship  with  DOT&E  and  other  organizations  involved  in  the  procurement  of  ACHs. 
We  also  analyzed  information  on  the  ACH  contracting  and  procurement  processes, 
requirements  and  specification,  oversight  at  test  and  manufacturing  facilities,  and  test 
implementation  plans.  Most  importantly,  we  analyzed  details  of  the  legacy  ACH  test 
protocols,  details  of  the  DOT&E  helmet  test  protocols,  the  rationale  for  its  established 
requirements,  and  the  organizations  involved  in  its  development. 

We  conducted  the  assessment  at  U.S.  Army  Test  and  Evaluation  Center  (ATEC)  in  Aberdeen, 
Maryland,  which  will  be  conducting  future  lightweight  ACH  testing.  The  objective  of  the 
assessment  was  to  understand  the  ACH  test  processes  and  set-up,  and  to  understand 
ATEC’s  input  to  the  DOT&E  helmet  test  protocols. 

We  also  performed  assessments  at  ACH  vendor  facilities.  The  objectives  of  the  assess¬ 
ments  were  to  understand  the  ACH  manufacturing  process  and  determine  the  impact  of 
the  new  DOT&E  test  protocols  to  the  vendors.  We  also  verified  to  determine  if  vendors 
provided  input  to  the  DOT&E  helmet  test  protocols. 

Use  of  Technical  Assistance 

The  Quantitative  Methods  Division  provided  support  and  statistical  expertise  in  the  anal¬ 
ysis  of  DOT&E’s  First  Article  Testing  protocol.  Specifically,  QMD  analyzed  the  Resistance 
to  Penetration  requirements  and  statistical  method  to  determine  acceptability  of  combat 
helmets. 
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Representative  Slaughter's  Letter  to  the  DoD  Office  of 
Inspector  General 


COMMITTEE  ON  RULES 
Ranking  Member 


WASHING  I  ON  OH  Id 

2469  Rayhurn  Building 
Washington,  D.C.  20515  3221 
{2021  225  3615 


district  oi  nets: 


3120  FEDERAL  BUILDING 
100  STATE  STREET 
ROCHESTER,  NY  14614 
(585)  232  4850 


465  MAIN  STREET,  SUITE  105 
BUFFALO.  NY  14203 
{7161  853  5813 


LOUISE  M.  SLAUGHTER 
CONGRESS  OF  THE  UNITED  STATES 
28TH  DISTRICT,  NEW  YORK 


640  Park  Placi 
NIAGARA  FALLS,  NY  14301 
(716)  202  1274 

Wi  iirv  1 1  lntp;//www. luuiso.houbu.gov 


June  26,  2012 


Honorable  Lynne  M.  Halbrooks 
Acting  Inspector  General 
Department  of  Defense 
400  Army  Navy  Drive 
Arlington,  VA  22202-4704 


Dear  Inspector  General  Halbrooks: 

I  write  to  you  today  to  request  that  you  review  the  Department  of  Defense’s  (DoD)  current  testing 
standards  for  Advanced  Combat  Helmets  (ACHs). 

Since  2006, 1  have  worked  extensively  with  the  Department’s  Inspector  General  to  ensure  that  our 
troops  are  equipped  with  the  best  available  body  armor.  1  know  you  share  my  belief  that 
America’s  servicemen  and  women  deserve  the  highest  quality  protective  armor  and  equipment 
when  they  are  in  harm’s  way  for  our  country.  As  the  Department  has  learned  from  past  experience, 
safe  armor  and  equipment  in  the  field  starts  with  comprehensive  and  effective  testing  procedures 
at  home. 

Government  employees  and  contractors  have  shared  concerns  with  my  office  that  a  recent 
modification  to  the  standard  for  ballistic  testing  for  ACHs  would  allow  up  to  17  penetrations  out 
of  240  test  shots.  The  result  of  that  standard  is  a  90%  probability  of  no  perforation  with  90% 
confidence  during  first  article  testing.  I  am  concerned  that  a  standard  that  allows  1 7  penetrations 
puts  contractors  above  the  need  to  produce  effective  and  consistent  equipment  for  the  men  and 
women  of  the  military. 

While  procurement  standards  have  improved  in  recent  years,  the  current  standard  poses  an 
unacceptably  high  risk  for  equipment  intended  to  protect  American  lives. 

In  2010,  the  Department  was  forced  to  recall  44,000  ACHs  after  it  was  determined  the  helmets  did 
not  meet  service  standards  due  to  questionable  manufacturing  processes.  In  the  field,  a  penetration 
to  a  combat  helmet  could  result  in  a  fatality  or  extremely  serious  injury,  which  is  why  the 
Department  should  continue  to  take  testing  and  performance  concerns  seriously. 

Our  goal  should  always  be  the  most  effective  equipment  possible  and  as  such,  I  strongly  urge  you 
to  review  ballistics-testing  procedures  currently  in  place  to  ensure  we  are  meeting  this  goal. 
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Thank  you  for  your  continued  commitment  to  the  men  and  woman  of  our  Armed  Forces.  I  am 
confident  that  you  have  our  troops’  best  interests  in  mind  and  will  continue  do  your  very  best  to 
improve  the  quality  standards  of  all  protective  body  armor  and  equipment.  I  have  also  enclosed  a 
copy  of  a  letter  to  Secretary  Panetta  for  your  review.  1  look  forward  to  your  response. 


Sincerely, 


I  nnicp  Mrlntnch  Silaiiohter 


Louise  McIntosh  Slaughter 
Member  of  Congress 


Cc:  Honorable  Leon  E.  Panetta,  Secretary  of  Defense 
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Clopper-Pearson  Method 

The  Army  Research  Laboratory  published  a  paper  on  March  2011  titled  “A  Comparison  of 
Various  Methods  Used  to  Determine  the  Sample  Size  Requirements  for  Meeting  a  90/90 
Reliability  Specification/  by  David  W.  This  paper  explains  the  Clopper-Pearson  Method 
as  follows: 


2.1  The  Clopper-Pearson  Method 

The  Clopper-Pearson  (1934)  method  for  binomial  confidence  intervals  is  popular  for  its  relative 
ease  to  calculate.  In  general,  the  confidence  interval  limits,  (LCP,  UCP ) ,  are  solutions  to  the 


N 


statements  ^  LlCP  (\  —  LCP)j 


=  f  -d  £ 


N 


Ulcp(l-  UCP  y  1  =  —  .  If  at  least  one  success 


and  one  failure  are  observed  among  the  samples,  both  endpoints  of  the  interval  can  be  expressed 
as  functions  of  percentiles  from  F  distributions.  For  an  LCB,  with  X  =  N ,  the  calculation 
simply  reduces  to  LCP  =  yfa  .  Clopper-Pearson  intervals  are  often  referred  to  as  “exact”  intervals 
since  they  are  derived  from  exact  probability  statements  and  not  any  distributional 
approximations.  As  such,  the  Clopper-Pearson  is  often  touted  in  introductory  statistics 
textbooks. 


To  satisfy  the  90/90  reliability  specification  using  the  Clopper-Pearson  method,  we  seek  the 

i/ 

minimum  value  of  N  which  satisfies  .90  <  A0/N  .  Taking  the  logarithm  of  both  sides  of  this 

inequality,  we  have  In  (.90)  <  — ^  ^  which  leads  to  the  solution  N  >  ln(.l)/ln(.9)  =  21.85  . 

Since  N  must  be  an  integer,  the  number  of  zero-failure  trials  required  to  meet  the  90/90 
specification  under  the  Clopper-Pearson  method  is  rounded  up  to  22. 
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Notice  of  Concern  and  Notice  of  Concern  Resolution 


DEC  07  2012 

MEMORANDUM  FOR  DEPARTMENT  OF  DEFENSE  (DoD)  INSPECTOR  GENERAL  (IG) 

SUBJECT:  Advanced  Combat  Helmet  (ACH)  Technical  Assessment  (Project  No.  D2012- 
DT0TAD-007)  Memorandum  for  Record 

DOT&E  and  Program  Executive  Office  (PEO)  Soldier  concur  with  the  attached 

Memorandum  of  Record  that  describes  the  topics  discussed  during  the  November  14, 2012 

meeting  between  the  DoD  IG  ACH  Technical  Assessment  Team,  DOT&E,  and  PEO  Soldier. 

DOT&E  and  PEO  Soldier  also  acknowledge  that  the  issues  identified  in  the  Notice  of  Concern 


J.  Michael  Gilmore 
Director 

Operational  Test  &  Evaluation 


Attachment: 
As  stated 
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DEPARTMENT  OF  DEFENSE 
4800  MARK  CENTER  DRIVE 
ALEXANDRIA,  VIRGINIA  22350-1500 


INSPECTOR  GENERAL 


DEC  0  5  2012  - 


MEMORANDUM  FOR  DIRECTOR,  OPERATIONAL  TEST  AND  EVALUATION 
U.S.  ARMY  PROGRAM  EXECUTIVE  OFFICE,  SOLDIER 

_ SUBJECT:. AdYanced_CombatJielmet  (ACH)JTechnical  Assessment  _  ... _ 

(Project  No.  D2012-  DTOTAD-0007) 

The  attached  Memorandum  for  Record  (TAB  A)  describes  the  topics  discussed  during 
our  meeting  held  on  November  14, 2012,  to  discuss  issues  identified  in  the  attached  Notice  of 
Concern  -  Advanced  Combat  Helmet  Technical  Assessment  (TAB  B).  Please  respond  with  your 
concurrence  with  the  documented  Memorandum  for  Record  and  acknowledge  that  the  issues 
identified  in  the  Notice  of  Concern  have  been  satisfactorily  resolved. 


If  you  have  any  questions,  please  contact  me  or  Ms.  Anna  Ferre  at  703-604-8931. 


Randolph  R.  Stone 
Deputy  Inspector  General 
Policy  and  Oversight 


Attachments: 
As  stated 
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Project  Title:  Advanced  Combat  Helmet  (ACH)  Technical  Assessment _ 

Project  Number:  D2012-DTQTAD-0QQ7Q0Q _  MFR  Author:  Anna  Ferre 


Location:  Pentagon  3E108B _  Date:  Wednesday,  November  14h  2012 

Purpose 

Continued  discussion  on  the  helmet  protocols. 


Attendees 

As  Stated  Below 


Action  Items/Documents  Requested 
Documents  Obtained 


Discussion 

Members  from  the  Department  of  Defense  Office  of  the  Inspector  General  (DoD  JG);  Office  of  the 
Director,  Operational  Test  and  Evaluation  (DOT&E);  and  the  Army’s  Program  Executive  Office 
(PEO),  Soldier,  met  on  November  14,  2012,  to  discuss  the  DoD  fG  team's  issues  developed  during 
their  technical  assessment  of  the  testing  of  the  Advanced  Combat  Helmet  (ACM)*  Representing 
DOT&E  was  the  Honorable  Dr.  J.  Michael  Gilmore,  Director,  QT&E;  representing  the  DoD  IG  was 
Mr,  Randolph  Stone,  Deputy  Inspector  General  for  Policy  and  Oversight;  and  representing  PEO 
Soldier  was  BG  Paul  Osirowski.  Also  attending  were  Mr,  Richard  Sayre  and  Mr,  Chris  Moosmann 
from  DOT&E;  Mr,  A I  Dopita,  Ms*  Anna  Ferre,  and  Mr.  Andrew  Fecehelm  from  the  DoD  IG;  and  ETC 
Frank  Lozano  from  PEO  Soldier.  The  attendees  addressed  the  following  topics; 

DOT&E  Authority 

The  DoD  IG  requested  further  clarification  of  DOT&E’s  authority  to  set  policies  pertaining  to 
personnel  protective  equipment  (PPE)  test  standards  and  the  process  for  which  test  protocols  are 
formally  coordinated  and  staffed  through  the  user  community's  policy  makers*  The  DoD  IG  wanted  to 
know  how  DOT&E  ensures  that  the  Services  have  an  opportunity  to  provide  input  to  PPE  policies* 

More  specifically,  the  DoD  IG  was  concerned  that  input  by  the  Service's  leadership  was  not  apparent, 
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DOT&E  stated  that  they  have  the  authority  to  set  policies  pertaining  to  PPE  test  standards  in 
accordance  with  the  authorities  given  to  the  Director  by  statute.  DOT&E  explained  that  test  standards 
are  a  policy  issue  formed  by  a  technical  understanding  of  both  the  testing  process  and  the  capabilities 
of  the  item  under  test.  PPE  testing  must  be  sufficiently  rigorous  and  based  on  appropriate  statistical 
principals  to  draw  valid  conclusions  about  performance  with  reasonable  confidence.  The  lack  of 
formal  requirements  documents  tor  PPE  {hard  armor,  helmets,  and  soft  armor)  led  DOT&E  to  publish 
the  protocols  to  establish  common  testing  and  performance  standards  for  these  items  as  items  overseen 
by  DOT&E  in  accordance  with  section  2366,  title  10,  United  States  Code  (10  U.S.C.  236 6).  DOT&E 
solicits  input  from  the  user  community  (including  program  offices,  test  agencies,  combat  developers, 
and  non-DoD  agencies  such  as  the  FBI  and  NIST)  with  regard  to  PPE  test  protocols. 

The  Director,  OT&E,  stated  he  will  provide  proposed  protocols  to  the  senior  leadership  of  the  affected 
Service(s)  for  comment.  Input  received  will  be  considered  and  adjudicated  by  the  Director,  The  DoD 
IG  agreed  that  providing  proposed  protocols  for  review  at  the  senior  leadership  level  would  help 
ensure  visibility  of  the  protocols  within  the  Services. 

DoD  IG  Issues  Developed  During  the  Technical  Assessment 

The  DoD  IG  acknowledged  that  the  new  helmet  protocols  am  an  improvement  from  legacy  helmet  test 
protocols  with  regard  to  increased  sample  size  and  statistical  integrity.  While  the  legacy  First  Article 
Test  (FAT)  protocols  tested  4  helmets,  the  new  protocol  tests  48  helmets  and  provide  data  that 
demonstrate  differences  in  performance  due  to  environmental  conditions*  helmet  sizes,  and  different 
vendor  designs.  However,  the  DoD  IG  identified  four  issues  as  a  result  of  documentation  and 
information  provided  by  DOT&E  and  PEO  Soldier, 

The  DoD  IG  team  summarized  the  issues  as: 

*  Lack  of  lie  1  met  performance  characterization  lor  the  ACH  protocol  before  it  was  approved, 

*  DGT&Ef$  apparent  desire  to  standardize  statistical  performance  measures  across  PPE. 

*  Lack  of  a  specific  policy  to  allow  test  protocol  modification  by  Service  technical  design 

authorities, 

*  The  statistical  implications  of  aggregating  test  results  across  environmental  conditions,  shot 

locations,  and  helmet  sizes. 

Helmet  Performance  Characterization 

The  DoD  IG  referenced  a  2010  U.S.  Army  Test  and  Evaluation  Center  (ATEC)  study  that 
characterized  the  body  armor  ballistic  performance  and  recommended  to  DOT&E  a  sample  size  and 
ballistic  performance  parameters  for  the  Lest  protocols  (90%  probability  of  no  penetration  [P(nP)]  with 
90%  confidence  level,  and  4%  Acceptable  Quality  Level.)  The  DoD  IG  was  concerned  that  a 
characterization  study  was  not  conducted  for  combat  helmets  prior  to  the  release  of  the  new  helmet  test 

2  of  5 


II.  July  25.  2012 


32 


DODIG-2013-079 


Appendices 


DOD  Office  of  the  Inspector  General 
TAD 

Memorandum  for  Record 

protocols. 

DOT&E  and  PEO  Soldier  committed  to  fully  characterize  the  performance  of  the  ACH  helmets. 
DOT&E  also  indicated  that  it,  along  with  the  Marine  Corps,  will  characterize  the  performance  ofthc 
Enhanced  Combat  Helmet  (ECH)  once  the  ECH  design  is  stable.  DOT&E  will  provide  the  details  oF 
the  plans  to  characterize  the  helmets  to  the  DoD  IG  team  when  the  respective  plans  are  approved  by 
DOT&E. 

Standardization  of  Protocols  Across  PPE 

The  DoD  IG  stated  that  there  arc  potential  issues  with  DOT&E's  plans  to  standardize  test  protocols 
across  all  PPE  categories  (hard  armor,  soft  armor,  and  helmet).  The  DoD  IG  asked  if  a  standard  Lest 
protocol  across  all  PPE  categories  is  appropriate  given  their  differences  in  material  and  manufacturing 
processes. 

DOT&E  considers  the  established  test  standards  for  hard  body  armor  plates  and  helmets  that  reflect  a 
common  metric  to  have  elements  that  are  generally  applicable  to  all  PPE,  Specifically,  the  standards 
for  PPE  testing  must  be  as  statistically  principled  and  rigorous  as  the  extant  standards  lor  helmets  and 
hard  body  armor  plates.  DOT&E  acknowledged  that  the  performance  metrics  incorporated  in  test 
standards  used  for  PPE  should  be  empirically  based  on  the  data  generated  by  characterization  studies 
conducted  on  the  individual  elements  of  PPE  using  the  appropriate  threats.  DOT&E  stated  that 
deviation  from  what  is  commonly  termed  a  “90/90  performance  metric"  could  be  appropriate,  based  on 
the  performance  revealed  by  these  characterization  studies  against  the  specified  threats.  Therefore,  the 
DoDIG  team  understood  that  DOT&E's  efforts  to  establish  standardized  protocols  does  not  necessarily 
mean  the  same  exact  performance  metrics  will  be  incorporated  in  the  testing  protocols  used  for  all 
categories  of  PPE;  rather,  those  metrics  will  be  standardized  by  PPE  category,  as  appropriate. 

Protoco  1  Modification 

The  DoD  IG  also  cited  issues  with  the  technical  design  authority's  ability  to  set  test  requirements 
above  the  current  test  protocols.  The  current  helmet  protocols  do  not  explicitly  allow  for  program 
managers  to  set  more  stringent  test  requirements.  If  DOT&E  continues  with  plans  to  standardize  test 
protocols  across  all  PPEs,  program  managers  are  not  able  to  implement  more  stringent  testing 
standards  as  appropriate. 

In  current  practice.  DOT&E  allows  protocols  to  be  changed  and,  in  particular,  made  more  stringent, 
provided  program  managers  submit  to  DOT&E  for  approval  a  request  that  is  well-justified.  This 
policy,  that  changes  to  protocols  are  possible  but  DOT&E  retains  the  authority  to  approve  such 
changes,  will  be  described  explicitly  in  protocol  updates  and  future  protocols. 
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Aggregation  of  Test  Results 

The  last  issue  pertains  to  the  aggregation  of  test  results  across  environmental  conditions,  shot 
locations,  and  helmet  sizes.  Using  this  evaluation  method,  it  is  possible  that  a  helmet  may  fail 
individually  for  ambient,  hot,  cold,  or  seawater  test  conditions,  yet  pass  if  the  test  results  are 
aggregated  and  calculated  using  the  Clapper- Pearson  method. 

DOT&E  agreed  and  stated  that  additional  statistical  analysis  will  be  conducted  beyond  that  specified 
in  the  protocol.  With  respect  to  aggregation  of  helmet  test  data,  DOT&E  will  analyze  whether 
significant  clusters  of  inadequate  performance  exist  in  both  the  backface  deformation  and  the 
penetration  data.  For  backface  deformation,  DOT&E  will  use  regression  analysis  to  lest  for  significant 
differences.  If  the  results  are  significant,  then  deformation  will  be  reported  separately  by  condition 
using  the  regression  model.  Failures  would  trigger  additional  testing,  sufficient  to  determine  the  90/90 
criteria  under  the  test  condilion(s)  of  interest.  DOT&E  will  use  either  logistic  regression  analysis  or 
two-sample  tests  of  proportions  to  analyze  the  probability  of  no  perforation  for  significant  clusters. 
Again,  significant  results  would  trigger  additional  testing  to  determine  whether  helmet  protocol 
requirements  were  met  by  condition.  DOT&E  will  issue  a  memorandum  describing  the  technical 
details  of  this  analysis  and  will  subsequently  incorporate  this  approach  in  a  formal  update  to  the 
existing  lest  protocol 

DOT&E  indicated  that  DGT&E  and  PEO  Soldier  will  adopt  this  approach  with  the  current  lightweight 
ACH  solicitation,  but  not  as  a  formal  change  to  the  test  protocol  specified  in  the  current  solicitation. 
Although  the  solicitation  will  not  be  changed,  DOl'&E  will  analyze  the  test  results  obtained  under  the 
current  solicitation  to  sec  if  significant  differences  (based  on  environment,  helmet  size,  or  shot 
location)  exist,  and  if  so,  conduct  additional  testing  to  then  determine  the  90/90  criteria  under  the  test 
condition  of  interest.  If,  following  that  additional  testing,  the  results  demonstrated  poor  performance, 
the  lightweight  ACH  design  will  not  be  fielded.  The  DoD  1G  team  indicated  this  approach  resolved 
the  issues  identified  in  the  Notice  of  Concern  and  did  not  express  any  need  to  revise  the  ACH 
solicitation  as  discussed  in  DOT&E's  October  12,  2012,  response  to  the  DoDIG  team's  Notice  of 
Concern. 

Further  Review 

DOT&E  has  contracted  with  the  National  Research  Council  (NRC)  to  conduct  an  external  review  of 
helmet  test  protocols  to  evaluate  the  adequacy  of  the  current  helmet  test  protocols  and  procedures.  The 
NRC  has  appointed  a  panel  of  distinguished  scientists,  statisticians,  and  subject  matter  experts  to 
accomplish  this  review.  DOT&E  will  inform  the  panel  on  all  of  the  previously  discussed  issues  and 
invite  the  DoD  IG  team  to  participate  in  the  panel’s  public  data  gathering  sessions.  The  work  of  the 
NRC  will  be  used  by  DOT&E  to  make  further  changes  to  the  existing  test  protocols  and  procedures  for 
combat  helmets,  as  appropriate.  DOT&E  will  inform  the  DoD  IG  of  all  of  the  NRC's  findings  and  any 
associated  changes  to  the  helmet  test  protocols  and  procedures. 
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DOD  Office  of  the  Inspector  General 
TAD 

Memorandum  for  Record 


Attachments 

Associated  Documents 

Q  Attending  Party  Review  Not  Required  (As  determined  by  TAD  PM  or  Supervisor} 
Attending  Party  Review 

I  agree  with  the  summary  of  this  meeting.  Q  Yes  Q  No 


Signature/ Date 


MFR  Approved: 


DGPITA  ALOIS.  1 090  3  S 


2744 


TAD  PM  or  Supervisor  Signature/Date 
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INSPECTOR  GENERAL 

DEPARTMENT  OF  DEFENSE 
4800  MARK  CENTER  DRIVE 
ALE  KANDRI A.  VIRGINIA  2235D-1S00 


SEP  1  2  M1Z 


MEMORANDUM  FOR  DIRECTOR,  OPERATIONAL  TEST  AND  EVALUATION 
PROGRAM  EXECUTIVE  OFFICE  SOLDIER 


SUBJECT:  Notice  of  Concern-Advanced  Combat  Helmet  Technical  Assessment 
(Project  No,  D20 1 2-DTOTAD-0007) 

References:  (a)  DOT&E  Memorandum*  “Standardisation  of  Combat  Helmel  Testing,” 
September  20, 201 1 

(b)  DOT&E  Memorandum,  "Standard  for  Lol  Acceptance  Ballistic  Testing  of 
Military  Combat  Helmets,”  May  4, 2012 

(c)  PEO  Soldier  Study,  “Advanced  Combat  Helmet  (ACH)  Resistance  to 
Penetration  Analysis,"  July  31,  2012 

We  are  issuing  this  Notice  of  Concern  to  inform  you  that  the  Department  of  Defense, 
Inspector  General  (DoD  IG)  team  identified  two  issue  dial  requires  your  attention*  During 
meetings  held  with  your  office  and  Office  of  the  Director,  Operational  Test  and  Evaluation 
(DOT&E)  at  Fort  Be! voir,  Virginia,  from  August  27  to  29, 2012,  your  team  provided  information 
that  indicates  that  the  historical  Advanced  Combat  Helmet  (ACH)  ballistic  performance  is 
significantly  higher  than  the  standards  defined  in  References  (a)  and  (b). 

During  our  meetings,  we  received  Reference  (c),  which  details  historical  ACH  test  data 
from  the  past  5  years.  Your  office  summarized  that  when  the  Cl opper-Pe arson  Method  is  applied 
to  the  aggregate  test  data,  the  ACH  Probability  of  No  Penetration  P(nP)  is  99%  with  a 
99%  Lower  Confidence  Level.  Furthermore,  you  concluded  that  “Currently  fielded  ACH  far 
exceeds  DOT&E’s  statistical  requirement  of  a  90%  Probability  of  no  Penetration  with  n  90% 
Lower  Confidence  Level  when  assessed  against  the  9mm  threat.” 

The  Acceptable  Quality  Level  (AQL)  detailed  in  Reference  (b)  is  another  area  of 
concern.  The  DOT&E-mandated  4%  AQL  (4  ballistic  defects  per  100  helmets)  for  lot 
acceptance  test  lowers  performance  expectations  because  past  AC1 1  lot  acceptance  lest  standards 
allow  for  zero  ballistic  defects. 

Our  concern  is  that  the  lightweight  ACH  solicitation,  scheduled  to  be  awarded  during  the 
first  quarter  of  fiscal  year  2013,  will  set  a  ballistic  performance  expectation  that  is  significantly 
lower  than  historical  capabilities.  This  contract  award  will  be  the  first  ACH  contract  to  include 
DOT&E's  ACH  lest  protocols  defining  a  P(nP)  of  at  least  90%  and  an  AQL  of  4%t 

The  DoD  IG  team’s  overall  conclusion  is  that  mandating  ballistic  performance 
expectations  that  are  significantly  lower  than  historical  capabilities  may  lead  to  a  reduction  in 
future  ACH  helmet  performance.  We  recommend  that  PEQ  Soldier  coordinate  with  the  Defense 
Logistics  Agency  (DLA)  to  assess  the  upcoming  solicitation  to  ensure  that  there  is  no  reduction 
in  helmet  performance.  We  also  recommend  that  DOT&E  coordinate  with  PEO  Soldier  to 
ensure  that  the  historical  test  data  are  taken  into  account  in  the  lest  protocol  so  that  future  helmet 
procurements  meet  or  exceed  historical  capabilities.  Finally,  we  recommend  that  DOT&E,  PEO 
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Soldier,  and  DLA  work  together  to  review  the  proposal  data  and  verify  that  ACH  ballistic  quality 
is  not  reduced. 

Please  provide  comments  that  state  whether  you  agree  or  disagree  with  the  issue  and 
recommendation.  If  you  agree,  describe  what  actions  you  have  taken  or  plan  to  take  to  address 
the  concerns  and  include  Ihe  completion  dates  of  your  actions.  If  you  disagree*  please  give 
specific  reasons  why  you  disagree  and  propose  alternative  action  if  appropriate.  Once  we 
complete  our  full  assessment,  we  will  provide  a  report  on  our  findings,  along  with  any  additional 
matters  that  might  come  to  our  attention. 

We  should  receive  your  comments  by  October  12,  2012.  Wc  normally  include  copies  of 
the  comments  in  the  report.  Jf  you  consider  any  matters  to  be  exempt  from  public  release,  you 
should  mark  them  clearly  for  Inspector  General  consideration. 

If  possible*  send  a  Microsoft  Word  (.doc)  Hie  and  a  portable  document  formal  (.pdf)  file 
containing  your  comments  to  anna,  ferre@dod  ig.mil.  Copies  of  your  comments  must  have  the 
actual  signature  of  the  authorizing  official  for  your  organization.  Wc  are  unable  to  accept  the 
/Signed/  symbol  in  place  of  the  actual  signature. 


We  appreciate  the  courtesies  extended  to  the  staff.  Please  direct  questions  to  Ms.  Anna 
Ferre  at  (703)604-8931, 


Randolph  R.  Stone 
Deputy  Inspector  General 
Policy  and  Oversight 


cc: 

Defense  Logistics  Agency 
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Appendix  E 

Office  of  the  Director,  Operational  Test  and  Evaluation 
and  Program  Executive  Office  Soldier  Comments 


APR  1  9  2013 

MEMORANDUM  FOR  DEPARTMENT  OF  DEFENSE  (DoD)  INSPECTOR  GENERAL  (IG) 

SUBJECT:  Director,  Operational  Test  and  Evaluation  and  Program  Executive  Officer  Soldier 
Comments  on  DoD  IG  Report:  Advanced  Combat  Helmet  (ACH)  Technical 
Assessment 

Reference:  DoD  IG  Memorandum,  “Advanced  Combat  Helmet  Technical  Assessment 
(Project  No.  D2012-DT0TAD-0007.000),”  April  1,  2013 

DOT&E  and  Program  Executive  Office  (PEO)  Soldier  have  reviewed  the  referenced 
report  and  agree  with  the  findings  and  recommendations.  This  response  provides  our  comments 
on  the  recommendations. 

Recommendation  for  Finding  A  (Origin  of  Resistance  to  Penetration 
Requirements):  Recommendation  that  DOT&E  and  PEO  Soldier  fully  characterize  the 
performance  of  all  helmet  designs  included  in  the  combat  helmet  test  protocols.  Performance 
characterization  should  consider  threat,  historical  test  data,  prototype  test  data,  and 
manufacturing  capabilities.  Based  on  helmet  performance  characterizations,  DOT&E  and  PEO 
Soldier  should  determine  if  modification  to  the  FAT  and  LAT  protocols  are  appropriate. 

Comment:  Characterization  of  fielded  ACH  will  rely  on  historical  test  data  since  the 
Army  does  not  intend  to  buy  additional  ACH  designs.  We  will  conduct  additional  testing 
to  characterize  helmets  not  currently  fielded,  such  as  the  Lightweight  ACH,  the  Enhanced 
Combat  Helmet  (ECH),  and  the  Soldier  Protective  System  Integrated  Head  Protection 
System  (SPS  IHPS).  This  characterization  will  be  based  on  a  risk  assessment  that 
identifies  and  documents  the  testing  that  provides  the  most  important  data,  since  it  is 
unrealistic  to  scope  a  characterization  effort  that  includes  every  desirable  test.  Based  on 
the  characterization  data,  DOT&E  will  determine  whether  FAT  and  LAT  protocols 
should  be  modified. 

Recommendation  for  Finding  B  (Acceptable  Quality  Level  Based  on  Safety 
Criticality):  Recommendation  that  in  addition  to  considerations  made  to  Government  risk, 
manufacturer  risk,  and  historical  LAT  data,  DOT&E  should  also  consider  the  safety  criticality  of 
the  helmet  in  establishing  the  AQL  requirement  for  LAT  RTP  in  future  protocol  revisions. 

Comment:  DOT&E  will  consider  helmet  safety  criticality  as  an  additional  consideration 
in  establishing  the  AQL  requirement  for  LAT  RTP  during  the  next  protocol  revision. 

The  next  protocol  revision  will  consider  recommendations  from  the  on-going  National 
Academies  review  of  helmet  test  protocols.  This  study  should  be  complete  by  December 
2013. 

Recommendation  for  Finding  C  (Coordination  of  Helmet  Test  Protocols): 

Recommendation  that  DOT&E  ensure  the  affected  organizations,  including  heads  of  the  Military 
Departments,  are  consulted  in  developing  the  protocols  and  have  an  opportunity  to  provide  input 
on  new  or  changed  test  protocols.  DOT&E  should  also  ensure  that  inputs  received  are 
documented  and  adjudicated. 
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Comment:  DOT&E  will  ensure  affected  organizations,  including  heads  of  Military 
Departments,  are  consulted  in  developing  new  or  changed  protocols.  Responses  will  be 
adjudicated  by  the  DOT&E  and  documented.  A  request  to  both  the  Secretary  of  the 
Army  and  Secretary  of  the  Navy  to  review  the  current  protocol  resulted  in  no 
recommendations  for  changes  to  the  protocol  by  both  Departments. 

Recommendation  that  PEO  Soldier  solicit  input  from  helmet  manufacturers  and  DCMA  on  new 
or  changed  test  protocols.  PEO  Soldier  should  also  ensure  that  inputs  received  are  documented 
and  adjudicated. 

Comment:  PEO  Soldier  will  solicit  feedback  from  our  industry  partners  as  well  as 
DCMA  prior  to  the  implementation  of  any  new  testing  protocol.  Industry  feedback 
together  with  feedback  from  many  various  engineers  working  for  PEO  Soldier  will  be 
reviewed  and  adjudicated.  All  adjudicated  feedback  will  be  provided  to  DOT&E  for 
review  and  may  warrant  the  continued  modification  of  existing  and  future  test  protocols 
and  standards. 

Recommendation  that  DOT&E  include  an  explicit  statement  in  future  protocol  revisions  that 
allows  program  managers  to  modify  the  test  protocols,  provided  that  program  managers  submit  a 
well-justified  request  for  approval. 

Comment:  The  next  protocol  revision  will  include  an  explicit  statement  that  permits 
program  managers  to  increase  the  test  protocol  stringency  following  approval  by  DOT&E 
of  a  well-justified  request  to  do  so. 

Recommendation  for  Finding  D  (Evaluation  of  Aggregate  Test  Results): 

Recommendation  that  DOT&E  and  PEO  Soldier  describe  the  method  of  identifying  and 
addressing  statistically  significant  differences  in  performance  due  to  environmental  conditions, 
helmet  sizes,  shot  locations,  and  different  vendor  designs  for  all  FAT  results  under  the  DOT&E 
helmet  test  protocol. 

Comment:  DOT&E  has  developed  procedures  for  identifying  and  addressing  statistically 
significant  differences  in  performance.  DOT&E  has  asked  the  National  Academies,  as 
part  of  its  review  of  the  helmet  test  protocols,  to  comment  on  the  adequacy  of  these 
procedures  to  identify  significant  differences  in  performance.  DOT&E  will  incorporate 
these  procedures  (as  revised  based  on  any  feedback  from  the  National  Academies)  in  the 
next  revision  of  the  protocol.  DOT&E  has  provided  a  copy  of  these  procedures  to  the 
DoD  IG  staff  for  information. 
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Recommendation  that  PEO  Soldier  and  DLA  consider  the  contractual  implications  of  not 
fielding  an  ACH  design  that  passes  FAT  yet  shows  significant  clusters  of  inadequate 
performance. 

Comment:  PEO  Soldier  is  working  with  DLA  to  develop  a  risk  mitigation  plan  to  limit 
contractual  implications  of  procuring  ACHs  that  pass  FAT  but  show  clusters  of 
inadequate  performance.  To  date,  based  on  historical  ACH  test  performance,  there  is 
nothing  to  lead  the  Product  Management  Office  to  believe  that  this  issue  would  occur; 
nevertheless,  PEO  Soldier  will  implement  risk  reduction  measures  to  ensure  clustering 
analysis  is  adequately  performed  and  only  helmets  meeting  the  entire  ballistic 


Program  Executive  Officer  Soldier 


Operational  Test  &  Evaluation 
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Acronyms  and  Abbreviations 


ACH 

Advanced  Combat  Helmet 

ADM 

Acquisition  Decision  Memorandum 

ANSI 

American  National  Standards  Institute 

AQL 

Acceptable  Quality  Level 

ASQ 

American  Society  for  Quality 

ATEC 

Army  Test  and  Evaluation  Center 

DCMA 

Defense  Contract  Management  Agency 

DLA  TS 

Defense  Logistics  Agency  Troop  Support 

DOT&E 

Office  of  the  Director,  Operational  Test  and  Evaluation 

ECH 

Enhanced  Combat  Helmet 

FAT 

First  Article  Testing 

FMJ  RN 

Full  Metal  Jacket  Round  Nose 

OIG 

Office  of  Inspector  General 

LAT 

Lot  Acceptance  Testing 

LCL 

Lower  Confidence  Level 

LFT&E 

Live  Fire  Test  and  Evaluation 

LWH 

Lightweight  Helmet 

NOC 

Notice  of  Concern 

ORD 

Operational  Requirements  Document 

PASGT 

Personnel  Armor  System,  Ground  Troops 

PEO 

Program  Executive  Office 

PM/SPE 

Product  Manager,  Soldier  Protective  Equipment 

P(nP) 

Probability  of  No  Penetration 

PPE 

Personnel  Protective  Equipment 

RTP 

Resistance  to  Penetration 
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Whistleblower  Protection 

U.S.  Department  of  Defense 


The  Whistleblower  Protection  Enhancement  Act  of  2012  requires 
the  Inspector  General  to  designate  a  Whistleblower  Protection 
Ombudsman  to  educate  agency  employees  about  prohibitions  on 
retaliation,  and  rights  and  remedies  against  retaliation  for  protected 
disclosures.  The  designated  ombudsman  is  the  DoD  IG  Director  for 
Whistleblowing  &  Transparency.  For  more  information  on  your  rights 
and  remedies  against  retaliation,  go  to  the  Whistleblower  webpage  at 
www.dodig.mil/programs/whistleblower. 


For  more  information  about  DoD  IG 
reports  or  activities,  please  contact  us: 

Congressional  Liaison 

Congressional@dodig.mil;  703.604.8324 

DoD  Hotline 

800.424.9098 

Media  Contact 

Public.Affairs@dodig.mil;  703.604.8324 

Monthly  Update 

dodigconnect-request@listserve.com 

Reports  Mailing  List 

dodig_report-request@listserve.com 


Twitter 

twitter.com/DoD_IG 


